libervia-backend: sat/memory/sqla.py annotate

annotate sat/memory/sqla.py @ 3586:5f65f4e9f8cb

plugin XEP-0060: getItems extra is now serialised dict

author	Goffi <goffi@goffi.org>
date	Wed, 28 Jul 2021 23:01:15 +0200
parents	16ade4ad63f3
children	7510648e8e3a

rev	line source
3537 f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	1 #!/usr/bin/env python3
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	2
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	3 # Libervia: an XMPP client
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	4 # Copyright (C) 2009-2021 Jérôme Poisson (goffi@goffi.org)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	5
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	6 # This program is free software: you can redistribute it and/or modify
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	7 # it under the terms of the GNU Affero General Public License as published by
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	8 # the Free Software Foundation, either version 3 of the License, or
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	9 # (at your option) any later version.
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	10
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	11 # This program is distributed in the hope that it will be useful,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	12 # but WITHOUT ANY WARRANTY; without even the implied warranty of
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	13 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	14 # GNU Affero General Public License for more details.
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	15
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	16 # You should have received a copy of the GNU Affero General Public License
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	17 # along with this program. If not, see <http://www.gnu.org/licenses/>.
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	18
3582 71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	19 import sys
3537 f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	20 import time
3582 71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	21 import asyncio
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	22 from asyncio.subprocess import PIPE
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	23 from pathlib import Path
3537 f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	24 from typing import Dict, List, Tuple, Iterable, Any, Callable, Optional
3582 71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	25 from sqlalchemy.ext.asyncio import AsyncSession, AsyncEngine, create_async_engine
3537 f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	26 from sqlalchemy.exc import IntegrityError, NoResultFound
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	27 from sqlalchemy.orm import sessionmaker, subqueryload, contains_eager
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	28 from sqlalchemy.future import select
3582 71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	29 from sqlalchemy.engine import Engine, Connection
3537 f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	30 from sqlalchemy import update, delete, and_, or_, event
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	31 from sqlalchemy.sql.functions import coalesce, sum as sum_
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	32 from sqlalchemy.dialects.sqlite import insert
3582 71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	33 from alembic import script as al_script, config as al_config
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	34 from alembic.runtime import migration as al_migration
3537 f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	35 from twisted.internet import defer
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	36 from twisted.words.protocols.jabber import jid
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	37 from sat.core.i18n import _
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	38 from sat.core import exceptions
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	39 from sat.core.log import getLogger
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	40 from sat.core.constants import Const as C
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	41 from sat.core.core_types import SatXMPPEntity
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	42 from sat.tools.utils import aio
3582 71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	43 from sat.memory import migration
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	44 from sat.memory import sqla_config
3537 f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	45 from sat.memory.sqla_mapping import (
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	46 NOT_IN_EXTRA,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	47 Base,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	48 Profile,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	49 Component,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	50 History,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	51 Message,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	52 Subject,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	53 Thread,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	54 ParamGen,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	55 ParamInd,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	56 PrivateGen,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	57 PrivateInd,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	58 PrivateGenBin,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	59 PrivateIndBin,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	60 File
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	61 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	62
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	63
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	64 log = getLogger(__name__)
3582 71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	65 migration_path = Path(migration.__file__).parent
3537 f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	66
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	67
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	68 @event.listens_for(Engine, "connect")
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	69 def set_sqlite_pragma(dbapi_connection, connection_record):
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	70 cursor = dbapi_connection.cursor()
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	71 cursor.execute("PRAGMA foreign_keys=ON")
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	72 cursor.close()
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	73
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	74
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	75 class Storage:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	76
3582 71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	77 def __init__(self):
3537 f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	78 self.initialized = defer.Deferred()
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	79 # we keep cache for the profiles (key: profile name, value: profile id)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	80 # profile id to name
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	81 self.profiles: Dict[int, str] = {}
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	82 # profile id to component entry point
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	83 self.components: Dict[int, str] = {}
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	84
3582 71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	85 async def migrateApply(self, *args: str, log_output: bool = False) -> None:
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	86 """Do a migration command
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	87
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	88 Commands are applied by running Alembic in a subprocess.
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	89 Arguments are alembic executables commands
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	90
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	91 @param log_output: manage stdout and stderr:
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	92 - if False, stdout and stderr are buffered, and logged only in case of error
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	93 - if True, stdout and stderr will be logged during the command execution
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	94 @raise exceptions.DatabaseError: something went wrong while running the
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	95 process
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	96 """
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	97 stdout, stderr = 2 * (None,) if log_output else 2 * (PIPE,)
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	98 proc = await asyncio.create_subprocess_exec(
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	99 sys.executable, "-m", "alembic", *args,
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	100 stdout=stdout, stderr=stderr, cwd=migration_path
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	101 )
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	102 log_out, log_err = await proc.communicate()
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	103 if proc.returncode != 0:
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	104 msg = _(
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	105 "Can't {operation} database (exit code {exit_code})"
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	106 ).format(
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	107 operation=args[0],
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	108 exit_code=proc.returncode
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	109 )
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	110 if log_out or log_err:
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	111 msg += f":\nstdout: {log_out.decode()}\nstderr: {log_err.decode()}"
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	112 log.error(msg)
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	113
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	114 raise exceptions.DatabaseError(msg)
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	115
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	116 async def createDB(self, engine: AsyncEngine, db_config: dict) -> None:
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	117 """Create a new database
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	118
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	119 The database is generated from SQLAlchemy model, then stamped by Alembic
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	120 """
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	121 # the dir may not exist if it's not the XDG recommended one
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	122 db_config["path"].parent.mkdir(0o700, True, True)
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	123 async with engine.begin() as conn:
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	124 await conn.run_sync(Base.metadata.create_all)
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	125
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	126 log.debug("stamping the database")
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	127 await self.migrateApply("stamp", "head")
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	128 log.debug("stamping done")
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	129
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	130 def _checkDBIsUpToDate(self, conn: Connection) -> bool:
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	131 al_ini_path = migration_path / "alembic.ini"
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	132 al_cfg = al_config.Config(al_ini_path)
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	133 directory = al_script.ScriptDirectory.from_config(al_cfg)
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	134 context = al_migration.MigrationContext.configure(conn)
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	135 return set(context.get_current_heads()) == set(directory.get_heads())
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	136
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	137 async def checkAndUpdateDB(self, engine: AsyncEngine, db_config: dict) -> None:
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	138 """Check that database is up-to-date, and update if necessary"""
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	139 async with engine.connect() as conn:
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	140 up_to_date = await conn.run_sync(self._checkDBIsUpToDate)
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	141 if up_to_date:
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	142 log.debug("Database is up-to-date")
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	143 else:
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	144 log.info("Database needs to be updated")
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	145 log.info("updating…")
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	146 await self.migrateApply("upgrade", "head", log_output=True)
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	147 log.info("Database is now up-to-date")
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	148
3537 f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	149 @aio
3582 71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	150 async def initialise(self) -> None:
3537 f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	151 log.info(_("Connecting database"))
3583 16ade4ad63f3 core (memory/sqla_mapping): fix some technical debt: Goffi <goffi@goffi.org> parents: 3582 diff changeset	152
3582 71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	153 db_config = sqla_config.getDbConfig()
3537 f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	154 engine = create_async_engine(
3582 71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	155 db_config["url"],
3537 f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	156 future=True
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	157 )
3583 16ade4ad63f3 core (memory/sqla_mapping): fix some technical debt: Goffi <goffi@goffi.org> parents: 3582 diff changeset	158
3582 71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	159 new_base = not db_config["path"].exists()
3537 f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	160 if new_base:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	161 log.info(_("The database is new, creating the tables"))
3582 71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	162 await self.createDB(engine, db_config)
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	163 else:
71516731d0aa core (memory/sqla): database migration using Alembic: Goffi <goffi@goffi.org> parents: 3537 diff changeset	164 await self.checkAndUpdateDB(engine, db_config)
3537 f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	165
3583 16ade4ad63f3 core (memory/sqla_mapping): fix some technical debt: Goffi <goffi@goffi.org> parents: 3582 diff changeset	166 self.session = sessionmaker(
16ade4ad63f3 core (memory/sqla_mapping): fix some technical debt: Goffi <goffi@goffi.org> parents: 3582 diff changeset	167 engine, expire_on_commit=False, class_=AsyncSession
16ade4ad63f3 core (memory/sqla_mapping): fix some technical debt: Goffi <goffi@goffi.org> parents: 3582 diff changeset	168 )
16ade4ad63f3 core (memory/sqla_mapping): fix some technical debt: Goffi <goffi@goffi.org> parents: 3582 diff changeset	169
3537 f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	170 async with self.session() as session:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	171 result = await session.execute(select(Profile))
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	172 for p in result.scalars():
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	173 self.profiles[p.name] = p.id
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	174 result = await session.execute(select(Component))
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	175 for c in result.scalars():
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	176 self.components[c.profile_id] = c.entry_point
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	177
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	178 self.initialized.callback(None)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	179
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	180 ## Profiles
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	181
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	182 def getProfilesList(self) -> List[str]:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	183 """"Return list of all registered profiles"""
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	184 return list(self.profiles.keys())
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	185
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	186 def hasProfile(self, profile_name: str) -> bool:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	187 """return True if profile_name exists
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	188
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	189 @param profile_name: name of the profile to check
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	190 """
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	191 return profile_name in self.profiles
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	192
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	193 def profileIsComponent(self, profile_name: str) -> bool:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	194 try:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	195 return self.profiles[profile_name] in self.components
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	196 except KeyError:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	197 raise exceptions.NotFound("the requested profile doesn't exists")
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	198
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	199 def getEntryPoint(self, profile_name: str) -> str:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	200 try:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	201 return self.components[self.profiles[profile_name]]
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	202 except KeyError:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	203 raise exceptions.NotFound("the requested profile doesn't exists or is not a component")
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	204
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	205 @aio
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	206 async def createProfile(self, name: str, component_ep: Optional[str] = None) -> None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	207 """Create a new profile
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	208
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	209 @param name: name of the profile
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	210 @param component: if not None, must point to a component entry point
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	211 """
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	212 async with self.session() as session:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	213 profile = Profile(name=name)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	214 async with session.begin():
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	215 session.add(profile)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	216 self.profiles[profile.id] = profile.name
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	217 if component_ep is not None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	218 async with session.begin():
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	219 component = Component(profile=profile, entry_point=component_ep)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	220 session.add(component)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	221 self.components[profile.id] = component_ep
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	222 return profile
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	223
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	224 @aio
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	225 async def deleteProfile(self, name: str) -> None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	226 """Delete profile
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	227
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	228 @param name: name of the profile
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	229 """
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	230 async with self.session() as session:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	231 result = await session.execute(select(Profile).where(Profile.name == name))
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	232 profile = result.scalar()
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	233 await session.delete(profile)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	234 await session.commit()
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	235 del self.profiles[profile.id]
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	236 if profile.id in self.components:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	237 del self.components[profile.id]
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	238 log.info(_("Profile {name!r} deleted").format(name = name))
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	239
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	240 ## Params
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	241
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	242 @aio
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	243 async def loadGenParams(self, params_gen: dict) -> None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	244 """Load general parameters
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	245
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	246 @param params_gen: dictionary to fill
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	247 """
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	248 log.debug(_("loading general parameters from database"))
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	249 async with self.session() as session:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	250 result = await session.execute(select(ParamGen))
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	251 for p in result.scalars():
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	252 params_gen[(p.category, p.name)] = p.value
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	253
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	254 @aio
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	255 async def loadIndParams(self, params_ind: dict, profile: str) -> None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	256 """Load individual parameters
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	257
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	258 @param params_ind: dictionary to fill
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	259 @param profile: a profile which must exist
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	260 """
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	261 log.debug(_("loading individual parameters from database"))
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	262 async with self.session() as session:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	263 result = await session.execute(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	264 select(ParamInd).where(ParamInd.profile_id == self.profiles[profile])
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	265 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	266 for p in result.scalars():
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	267 params_ind[(p.category, p.name)] = p.value
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	268
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	269 @aio
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	270 async def getIndParam(self, category: str, name: str, profile: str) -> Optional[str]:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	271 """Ask database for the value of one specific individual parameter
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	272
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	273 @param category: category of the parameter
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	274 @param name: name of the parameter
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	275 @param profile: %(doc_profile)s
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	276 """
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	277 async with self.session() as session:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	278 result = await session.execute(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	279 select(ParamInd.value)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	280 .filter_by(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	281 category=category,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	282 name=name,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	283 profile_id=self.profiles[profile]
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	284 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	285 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	286 return result.scalar_one_or_none()
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	287
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	288 @aio
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	289 async def getIndParamValues(self, category: str, name: str) -> Dict[str, str]:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	290 """Ask database for the individual values of a parameter for all profiles
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	291
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	292 @param category: category of the parameter
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	293 @param name: name of the parameter
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	294 @return dict: profile => value map
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	295 """
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	296 async with self.session() as session:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	297 result = await session.execute(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	298 select(ParamInd)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	299 .filter_by(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	300 category=category,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	301 name=name
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	302 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	303 .options(subqueryload(ParamInd.profile))
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	304 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	305 return {param.profile.name: param.value for param in result.scalars()}
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	306
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	307 @aio
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	308 async def setGenParam(self, category: str, name: str, value: Optional[str]) -> None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	309 """Save the general parameters in database
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	310
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	311 @param category: category of the parameter
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	312 @param name: name of the parameter
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	313 @param value: value to set
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	314 """
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	315 async with self.session() as session:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	316 stmt = insert(ParamGen).values(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	317 category=category,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	318 name=name,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	319 value=value
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	320 ).on_conflict_do_update(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	321 index_elements=(ParamGen.category, ParamGen.name),
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	322 set_={
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	323 ParamGen.value: value
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	324 }
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	325 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	326 await session.execute(stmt)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	327 await session.commit()
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	328
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	329 @aio
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	330 async def setIndParam(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	331 self,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	332 category:str,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	333 name: str,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	334 value: Optional[str],
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	335 profile: str
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	336 ) -> None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	337 """Save the individual parameters in database
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	338
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	339 @param category: category of the parameter
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	340 @param name: name of the parameter
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	341 @param value: value to set
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	342 @param profile: a profile which must exist
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	343 """
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	344 async with self.session() as session:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	345 stmt = insert(ParamInd).values(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	346 category=category,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	347 name=name,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	348 profile_id=self.profiles[profile],
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	349 value=value
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	350 ).on_conflict_do_update(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	351 index_elements=(ParamInd.category, ParamInd.name, ParamInd.profile_id),
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	352 set_={
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	353 ParamInd.value: value
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	354 }
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	355 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	356 await session.execute(stmt)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	357 await session.commit()
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	358
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	359 def _jid_filter(self, jid_: jid.JID, dest: bool = False):
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	360 """Generate condition to filter on a JID, using relevant columns
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	361
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	362 @param dest: True if it's the destinee JID, otherwise it's the source one
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	363 @param jid_: JID to filter by
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	364 """
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	365 if jid_.resource:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	366 if dest:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	367 return and_(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	368 History.dest == jid_.userhost(),
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	369 History.dest_res == jid_.resource
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	370 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	371 else:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	372 return and_(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	373 History.source == jid_.userhost(),
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	374 History.source_res == jid_.resource
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	375 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	376 else:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	377 if dest:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	378 return History.dest == jid_.userhost()
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	379 else:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	380 return History.source == jid_.userhost()
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	381
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	382 @aio
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	383 async def historyGet(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	384 self,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	385 from_jid: Optional[jid.JID],
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	386 to_jid: Optional[jid.JID],
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	387 limit: Optional[int] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	388 between: bool = True,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	389 filters: Optional[Dict[str, str]] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	390 profile: Optional[str] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	391 ) -> List[Tuple[
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	392 str, int, str, str, Dict[str, str], Dict[str, str], str, str, str]
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	393 ]:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	394 """Retrieve messages in history
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	395
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	396 @param from_jid: source JID (full, or bare for catchall)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	397 @param to_jid: dest JID (full, or bare for catchall)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	398 @param limit: maximum number of messages to get:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	399 - 0 for no message (returns the empty list)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	400 - None for unlimited
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	401 @param between: confound source and dest (ignore the direction)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	402 @param filters: pattern to filter the history results
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	403 @return: list of messages as in [messageNew], minus the profile which is already
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	404 known.
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	405 """
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	406 # we have to set a default value to profile because it's last argument
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	407 # and thus follow other keyword arguments with default values
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	408 # but None should not be used for it
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	409 assert profile is not None
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	410 if limit == 0:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	411 return []
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	412 if filters is None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	413 filters = {}
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	414
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	415 stmt = (
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	416 select(History)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	417 .filter_by(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	418 profile_id=self.profiles[profile]
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	419 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	420 .outerjoin(History.messages)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	421 .outerjoin(History.subjects)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	422 .outerjoin(History.thread)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	423 .options(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	424 contains_eager(History.messages),
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	425 contains_eager(History.subjects),
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	426 contains_eager(History.thread),
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	427 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	428 .order_by(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	429 # timestamp may be identical for 2 close messages (specially when delay is
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	430 # used) that's why we order ties by received_timestamp. We'll reverse the
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	431 # order when returning the result. We use DESC here so LIMIT keep the last
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	432 # messages
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	433 History.timestamp.desc(),
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	434 History.received_timestamp.desc()
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	435 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	436 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	437
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	438
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	439 if not from_jid and not to_jid:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	440 # no jid specified, we want all one2one communications
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	441 pass
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	442 elif between:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	443 if not from_jid or not to_jid:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	444 # we only have one jid specified, we check all messages
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	445 # from or to this jid
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	446 jid_ = from_jid or to_jid
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	447 stmt = stmt.where(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	448 or_(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	449 self._jid_filter(jid_),
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	450 self._jid_filter(jid_, dest=True)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	451 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	452 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	453 else:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	454 # we have 2 jids specified, we check all communications between
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	455 # those 2 jids
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	456 stmt = stmt.where(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	457 or_(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	458 and_(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	459 self._jid_filter(from_jid),
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	460 self._jid_filter(to_jid, dest=True),
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	461 ),
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	462 and_(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	463 self._jid_filter(to_jid),
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	464 self._jid_filter(from_jid, dest=True),
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	465 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	466 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	467 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	468 else:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	469 # we want one communication in specific direction (from somebody or
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	470 # to somebody).
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	471 if from_jid is not None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	472 stmt = stmt.where(self._jid_filter(from_jid))
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	473 if to_jid is not None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	474 stmt = stmt.where(self._jid_filter(to_jid, dest=True))
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	475
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	476 if filters:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	477 if 'timestamp_start' in filters:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	478 stmt = stmt.where(History.timestamp >= float(filters['timestamp_start']))
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	479 if 'before_uid' in filters:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	480 # orignially this query was using SQLITE's rowid. This has been changed
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	481 # to use coalesce(received_timestamp, timestamp) to be SQL engine independant
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	482 stmt = stmt.where(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	483 coalesce(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	484 History.received_timestamp,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	485 History.timestamp
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	486 ) < (
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	487 select(coalesce(History.received_timestamp, History.timestamp))
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	488 .filter_by(uid=filters["before_uid"])
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	489 ).scalar_subquery()
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	490 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	491 if 'body' in filters:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	492 # TODO: use REGEXP (function to be defined) instead of GLOB: https://www.sqlite.org/lang_expr.html
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	493 stmt = stmt.where(Message.message.like(f"%{filters['body']}%"))
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	494 if 'search' in filters:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	495 search_term = f"%{filters['search']}%"
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	496 stmt = stmt.where(or_(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	497 Message.message.like(search_term),
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	498 History.source_res.like(search_term)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	499 ))
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	500 if 'types' in filters:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	501 types = filters['types'].split()
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	502 stmt = stmt.where(History.type.in_(types))
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	503 if 'not_types' in filters:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	504 types = filters['not_types'].split()
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	505 stmt = stmt.where(History.type.not_in(types))
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	506 if 'last_stanza_id' in filters:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	507 # this request get the last message with a "stanza_id" that we
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	508 # have in history. This is mainly used to retrieve messages sent
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	509 # while we were offline, using MAM (XEP-0313).
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	510 if (filters['last_stanza_id'] is not True
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	511 or limit != 1):
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	512 raise ValueError("Unexpected values for last_stanza_id filter")
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	513 stmt = stmt.where(History.stanza_id.is_not(None))
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	514
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	515 if limit is not None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	516 stmt = stmt.limit(limit)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	517
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	518 async with self.session() as session:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	519 result = await session.execute(stmt)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	520
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	521 result = result.scalars().unique().all()
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	522 result.reverse()
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	523 return [h.as_tuple() for h in result]
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	524
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	525 @aio
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	526 async def addToHistory(self, data: dict, profile: str) -> None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	527 """Store a new message in history
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	528
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	529 @param data: message data as build by SatMessageProtocol.onMessage
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	530 """
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	531 extra = {k: v for k, v in data["extra"].items() if k not in NOT_IN_EXTRA}
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	532 messages = [Message(message=mess, language=lang)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	533 for lang, mess in data["message"].items()]
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	534 subjects = [Subject(subject=mess, language=lang)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	535 for lang, mess in data["subject"].items()]
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	536 if "thread" in data["extra"]:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	537 thread = Thread(thread_id=data["extra"]["thread"],
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	538 parent_id=data["extra"].get["thread_parent"])
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	539 else:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	540 thread = None
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	541 try:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	542 async with self.session() as session:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	543 async with session.begin():
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	544 session.add(History(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	545 uid=data["uid"],
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	546 stanza_id=data["extra"].get("stanza_id"),
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	547 update_uid=data["extra"].get("update_uid"),
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	548 profile_id=self.profiles[profile],
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	549 source_jid=data["from"],
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	550 dest_jid=data["to"],
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	551 timestamp=data["timestamp"],
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	552 received_timestamp=data.get("received_timestamp"),
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	553 type=data["type"],
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	554 extra=extra,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	555 messages=messages,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	556 subjects=subjects,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	557 thread=thread,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	558 ))
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	559 except IntegrityError as e:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	560 if "unique" in str(e.orig).lower():
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	561 log.debug(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	562 f"message {data['uid']!r} is already in history, not storing it again"
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	563 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	564 else:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	565 log.error(f"Can't store message {data['uid']!r} in history: {e}")
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	566 except Exception as e:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	567 log.critical(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	568 f"Can't store message, unexpected exception (uid: {data['uid']}): {e}"
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	569 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	570
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	571 ## Private values
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	572
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	573 def _getPrivateClass(self, binary, profile):
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	574 """Get ORM class to use for private values"""
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	575 if profile is None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	576 return PrivateGenBin if binary else PrivateGen
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	577 else:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	578 return PrivateIndBin if binary else PrivateInd
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	579
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	580
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	581 @aio
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	582 async def getPrivates(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	583 self,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	584 namespace:str,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	585 keys: Optional[Iterable[str]] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	586 binary: bool = False,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	587 profile: Optional[str] = None
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	588 ) -> Dict[str, Any]:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	589 """Get private value(s) from databases
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	590
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	591 @param namespace: namespace of the values
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	592 @param keys: keys of the values to get None to get all keys/values
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	593 @param binary: True to deserialise binary values
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	594 @param profile: profile to use for individual values
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	595 None to use general values
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	596 @return: gotten keys/values
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	597 """
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	598 if keys is not None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	599 keys = list(keys)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	600 log.debug(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	601 f"getting {'general' if profile is None else 'individual'}"
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	602 f"{' binary' if binary else ''} private values from database for namespace "
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	603 f"{namespace}{f' with keys {keys!r}' if keys is not None else ''}"
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	604 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	605 cls = self._getPrivateClass(binary, profile)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	606 stmt = select(cls).filter_by(namespace=namespace)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	607 if keys:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	608 stmt = stmt.where(cls.key.in_(list(keys)))
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	609 if profile is not None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	610 stmt = stmt.filter_by(profile_id=self.profiles[profile])
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	611 async with self.session() as session:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	612 result = await session.execute(stmt)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	613 return {p.key: p.value for p in result.scalars()}
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	614
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	615 @aio
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	616 async def setPrivateValue(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	617 self,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	618 namespace: str,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	619 key:str,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	620 value: Any,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	621 binary: bool = False,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	622 profile: Optional[str] = None
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	623 ) -> None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	624 """Set a private value in database
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	625
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	626 @param namespace: namespace of the values
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	627 @param key: key of the value to set
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	628 @param value: value to set
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	629 @param binary: True if it's a binary values
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	630 binary values need to be serialised, used for everything but strings
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	631 @param profile: profile to use for individual value
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	632 if None, it's a general value
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	633 """
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	634 cls = self._getPrivateClass(binary, profile)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	635
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	636 values = {
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	637 "namespace": namespace,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	638 "key": key,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	639 "value": value
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	640 }
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	641 index_elements = [cls.namespace, cls.key]
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	642
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	643 if profile is not None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	644 values["profile_id"] = self.profiles[profile]
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	645 index_elements.append(cls.profile_id)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	646
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	647 async with self.session() as session:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	648 await session.execute(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	649 insert(cls).values(**values).on_conflict_do_update(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	650 index_elements=index_elements,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	651 set_={
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	652 cls.value: value
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	653 }
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	654 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	655 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	656 await session.commit()
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	657
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	658 @aio
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	659 async def delPrivateValue(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	660 self,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	661 namespace: str,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	662 key: str,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	663 binary: bool = False,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	664 profile: Optional[str] = None
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	665 ) -> None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	666 """Delete private value from database
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	667
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	668 @param category: category of the privateeter
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	669 @param key: key of the private value
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	670 @param binary: True if it's a binary values
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	671 @param profile: profile to use for individual value
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	672 if None, it's a general value
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	673 """
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	674 cls = self._getPrivateClass(binary, profile)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	675
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	676 stmt = delete(cls).filter_by(namespace=namespace, key=key)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	677
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	678 if profile is not None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	679 stmt = stmt.filter_by(profile_id=self.profiles[profile])
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	680
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	681 async with self.session() as session:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	682 await session.execute(stmt)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	683 await session.commit()
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	684
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	685 @aio
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	686 async def delPrivateNamespace(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	687 self,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	688 namespace: str,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	689 binary: bool = False,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	690 profile: Optional[str] = None
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	691 ) -> None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	692 """Delete all data from a private namespace
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	693
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	694 Be really cautious when you use this method, as all data with given namespace are
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	695 removed.
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	696 Params are the same as for delPrivateValue
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	697 """
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	698 cls = self._getPrivateClass(binary, profile)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	699
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	700 stmt = delete(cls).filter_by(namespace=namespace)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	701
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	702 if profile is not None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	703 stmt = stmt.filter_by(profile_id=self.profiles[profile])
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	704
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	705 async with self.session() as session:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	706 await session.execute(stmt)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	707 await session.commit()
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	708
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	709 ## Files
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	710
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	711 @aio
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	712 async def getFiles(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	713 self,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	714 client: Optional[SatXMPPEntity],
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	715 file_id: Optional[str] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	716 version: Optional[str] = '',
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	717 parent: Optional[str] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	718 type_: Optional[str] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	719 file_hash: Optional[str] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	720 hash_algo: Optional[str] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	721 name: Optional[str] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	722 namespace: Optional[str] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	723 mime_type: Optional[str] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	724 public_id: Optional[str] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	725 owner: Optional[jid.JID] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	726 access: Optional[dict] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	727 projection: Optional[List[str]] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	728 unique: bool = False
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	729 ) -> List[dict]:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	730 """Retrieve files with with given filters
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	731
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	732 @param file_id: id of the file
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	733 None to ignore
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	734 @param version: version of the file
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	735 None to ignore
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	736 empty string to look for current version
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	737 @param parent: id of the directory containing the files
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	738 None to ignore
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	739 empty string to look for root files/directories
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	740 @param projection: name of columns to retrieve
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	741 None to retrieve all
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	742 @param unique: if True will remove duplicates
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	743 other params are the same as for [setFile]
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	744 @return: files corresponding to filters
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	745 """
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	746 if projection is None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	747 projection = [
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	748 'id', 'version', 'parent', 'type', 'file_hash', 'hash_algo', 'name',
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	749 'size', 'namespace', 'media_type', 'media_subtype', 'public_id',
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	750 'created', 'modified', 'owner', 'access', 'extra'
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	751 ]
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	752
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	753 stmt = select(*[getattr(File, f) for f in projection])
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	754
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	755 if unique:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	756 stmt = stmt.distinct()
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	757
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	758 if client is not None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	759 stmt = stmt.filter_by(profile_id=self.profiles[client.profile])
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	760 else:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	761 if public_id is None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	762 raise exceptions.InternalError(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	763 "client can only be omitted when public_id is set"
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	764 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	765 if file_id is not None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	766 stmt = stmt.filter_by(id=file_id)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	767 if version is not None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	768 stmt = stmt.filter_by(version=version)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	769 if parent is not None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	770 stmt = stmt.filter_by(parent=parent)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	771 if type_ is not None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	772 stmt = stmt.filter_by(type=type_)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	773 if file_hash is not None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	774 stmt = stmt.filter_by(file_hash=file_hash)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	775 if hash_algo is not None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	776 stmt = stmt.filter_by(hash_algo=hash_algo)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	777 if name is not None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	778 stmt = stmt.filter_by(name=name)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	779 if namespace is not None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	780 stmt = stmt.filter_by(namespace=namespace)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	781 if mime_type is not None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	782 if '/' in mime_type:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	783 media_type, media_subtype = mime_type.split("/", 1)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	784 stmt = stmt.filter_by(media_type=media_type, media_subtype=media_subtype)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	785 else:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	786 stmt = stmt.filter_by(media_type=mime_type)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	787 if public_id is not None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	788 stmt = stmt.filter_by(public_id=public_id)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	789 if owner is not None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	790 stmt = stmt.filter_by(owner=owner)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	791 if access is not None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	792 raise NotImplementedError('Access check is not implemented yet')
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	793 # a JSON comparison is needed here
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	794
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	795 async with self.session() as session:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	796 result = await session.execute(stmt)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	797
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	798 return [dict(r) for r in result]
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	799
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	800 @aio
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	801 async def setFile(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	802 self,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	803 client: SatXMPPEntity,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	804 name: str,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	805 file_id: str,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	806 version: str = "",
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	807 parent: str = "",
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	808 type_: str = C.FILE_TYPE_FILE,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	809 file_hash: Optional[str] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	810 hash_algo: Optional[str] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	811 size: int = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	812 namespace: Optional[str] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	813 mime_type: Optional[str] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	814 public_id: Optional[str] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	815 created: Optional[float] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	816 modified: Optional[float] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	817 owner: Optional[jid.JID] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	818 access: Optional[dict] = None,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	819 extra: Optional[dict] = None
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	820 ) -> None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	821 """Set a file metadata
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	822
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	823 @param client: client owning the file
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	824 @param name: name of the file (must not contain "/")
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	825 @param file_id: unique id of the file
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	826 @param version: version of this file
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	827 @param parent: id of the directory containing this file
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	828 Empty string if it is a root file/directory
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	829 @param type_: one of:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	830 - file
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	831 - directory
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	832 @param file_hash: unique hash of the payload
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	833 @param hash_algo: algorithm used for hashing the file (usually sha-256)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	834 @param size: size in bytes
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	835 @param namespace: identifier (human readable is better) to group files
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	836 for instance, namespace could be used to group files in a specific photo album
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	837 @param mime_type: media type of the file, or None if not known/guessed
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	838 @param public_id: ID used to server the file publicly via HTTP
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	839 @param created: UNIX time of creation
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	840 @param modified: UNIX time of last modification, or None to use created date
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	841 @param owner: jid of the owner of the file (mainly useful for component)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	842 @param access: serialisable dictionary with access rules. See [memory.memory] for details
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	843 @param extra: serialisable dictionary of any extra data
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	844 will be encoded to json in database
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	845 """
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	846 if mime_type is None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	847 media_type = media_subtype = None
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	848 elif '/' in mime_type:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	849 media_type, media_subtype = mime_type.split('/', 1)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	850 else:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	851 media_type, media_subtype = mime_type, None
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	852
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	853 async with self.session() as session:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	854 async with session.begin():
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	855 session.add(File(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	856 id=file_id,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	857 version=version.strip(),
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	858 parent=parent,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	859 type=type_,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	860 file_hash=file_hash,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	861 hash_algo=hash_algo,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	862 name=name,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	863 size=size,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	864 namespace=namespace,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	865 media_type=media_type,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	866 media_subtype=media_subtype,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	867 public_id=public_id,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	868 created=time.time() if created is None else created,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	869 modified=modified,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	870 owner=owner,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	871 access=access,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	872 extra=extra,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	873 profile_id=self.profiles[client.profile]
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	874 ))
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	875
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	876 @aio
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	877 async def fileGetUsedSpace(self, client: SatXMPPEntity, owner: jid.JID) -> int:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	878 async with self.session() as session:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	879 result = await session.execute(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	880 select(sum_(File.size)).filter_by(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	881 owner=owner,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	882 type=C.FILE_TYPE_FILE,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	883 profile_id=self.profiles[client.profile]
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	884 ))
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	885 return result.scalar_one_or_none() or 0
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	886
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	887 @aio
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	888 async def fileDelete(self, file_id: str) -> None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	889 """Delete file metadata from the database
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	890
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	891 @param file_id: id of the file to delete
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	892 NOTE: file itself must still be removed, this method only handle metadata in
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	893 database
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	894 """
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	895 async with self.session() as session:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	896 await session.execute(delete(File).filter_by(id=file_id))
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	897 await session.commit()
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	898
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	899 @aio
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	900 async def fileUpdate(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	901 self,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	902 file_id: str,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	903 column: str,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	904 update_cb: Callable[[dict], None]
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	905 ) -> None:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	906 """Update a column value using a method to avoid race conditions
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	907
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	908 the older value will be retrieved from database, then update_cb will be applied to
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	909 update it, and file will be updated checking that older value has not been changed
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	910 meanwhile by an other user. If it has changed, it tries again a couple of times
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	911 before failing
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	912 @param column: column name (only "access" or "extra" are allowed)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	913 @param update_cb: method to update the value of the colum
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	914 the method will take older value as argument, and must update it in place
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	915 update_cb must not care about serialization,
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	916 it get the deserialized data (i.e. a Python object) directly
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	917 @raise exceptions.NotFound: there is not file with this id
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	918 """
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	919 if column not in ('access', 'extra'):
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	920 raise exceptions.InternalError('bad column name')
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	921 orm_col = getattr(File, column)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	922
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	923 for i in range(5):
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	924 async with self.session() as session:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	925 try:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	926 value = (await session.execute(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	927 select(orm_col).filter_by(id=file_id)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	928 )).scalar_one()
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	929 except NoResultFound:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	930 raise exceptions.NotFound
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	931 update_cb(value)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	932 stmt = update(orm_col).filter_by(id=file_id)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	933 if not value:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	934 # because JsonDefaultDict convert NULL to an empty dict, we have to
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	935 # test both for empty dict and None when we have and empty dict
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	936 stmt = stmt.where((orm_col == None) \| (orm_col == value))
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	937 else:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	938 stmt = stmt.where(orm_col == value)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	939 result = await session.execute(stmt)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	940 await session.commit()
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	941
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	942 if result.rowcount == 1:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	943 break
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	944
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	945 log.warning(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	946 _("table not updated, probably due to race condition, trying again "
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	947 "({tries})").format(tries=i+1)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	948 )
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	949
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	950 else:
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	951 raise exceptions.DatabaseError(
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	952 _("Can't update file {file_id} due to race condition")
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	953 .format(file_id=file_id)
f9a5b810f14d core (memory/storage): backend storage is now based on SQLAlchemy Goffi <goffi@goffi.org> parents: diff changeset	954 )

Mercurial > libervia-backend

annotate sat/memory/sqla.py @ 3586:5f65f4e9f8cb