Mercurial > libervia-backend

annotate sat/memory/migration/env.py @ 4002:5245b675f7ad

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
plugin XEP-0313: don't wait for MAM to be retrieved in connection workflow: MAM retrieval can be long, and can be done after connection, message just need to be sorted when being inserted (i.e. frontends must do insort). To avoid blocking connection for too long and result in bad UX and timeout risk, one2one MAM message are not retrieved in background.
author Goffi <goffi@goffi.org>
date Fri, 10 Mar 2023 17:22:45 +0100
parents 54c249ec35ce
children 524856bd7b19
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3582
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
1 import asyncio
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
2 from logging.config import fileConfig
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
3 from sqlalchemy import pool
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
4 from sqlalchemy.ext.asyncio import create_async_engine
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
5 from alembic import context
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
6 from sat.memory import sqla_config
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
7 from sat.memory.sqla_mapping import Base
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
8
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
9 # this is the Alembic Config object, which provides
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
10 # access to the values within the .ini file in use.
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
11 config = context.config
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
12
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
13 # Interpret the config file for Python logging.
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
14 # This line sets up loggers basically.
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
15 fileConfig(config.config_file_name)
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
16
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
17 # add your model's MetaData object here
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
18 # for 'autogenerate' support
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
19 # from myapp import mymodel
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
20 # target_metadata = mymodel.Base.metadata
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
21 target_metadata = Base.metadata
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
22
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
23 # other values from the config, defined by the needs of env.py,
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
24 # can be acquired:
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
25 # my_important_option = config.get_main_option("my_important_option")
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
26 # ... etc.
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
27
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
28
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
29 def run_migrations_offline():
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
30 """Run migrations in 'offline' mode.
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
31
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
32 This configures the context with just a URL
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
33 and not an Engine, though an Engine is acceptable
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
34 here as well. By skipping the Engine creation
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
35 we don't even need a DBAPI to be available.
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
36
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
37 Calls to context.execute() here emit the given string to the
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
38 script output.
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
39
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
40 """
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
41 db_config = sqla_config.getDbConfig()
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
42 context.configure(
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
43 url=db_config["url"],
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
44 target_metadata=target_metadata,
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
45 literal_binds=True,
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
46 dialect_opts={"paramstyle": "named"},
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
47 )
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
48
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
49 with context.begin_transaction():
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
50 context.run_migrations()
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
51
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
52
3743
54c249ec35ce core (memory/migration): ignore FTS table when autogenerating script for migration:
Goffi <goffi@goffi.org>
parents: 3582
diff changeset
53 def include_name(name, type_, parent_names):
54c249ec35ce core (memory/migration): ignore FTS table when autogenerating script for migration:
Goffi <goffi@goffi.org>
parents: 3582
diff changeset
54 if type_ == "table":
54c249ec35ce core (memory/migration): ignore FTS table when autogenerating script for migration:
Goffi <goffi@goffi.org>
parents: 3582
diff changeset
55 if name.startswith("pubsub_items_fts"):
54c249ec35ce core (memory/migration): ignore FTS table when autogenerating script for migration:
Goffi <goffi@goffi.org>
parents: 3582
diff changeset
56 return False
54c249ec35ce core (memory/migration): ignore FTS table when autogenerating script for migration:
Goffi <goffi@goffi.org>
parents: 3582
diff changeset
57 return True
54c249ec35ce core (memory/migration): ignore FTS table when autogenerating script for migration:
Goffi <goffi@goffi.org>
parents: 3582
diff changeset
58
54c249ec35ce core (memory/migration): ignore FTS table when autogenerating script for migration:
Goffi <goffi@goffi.org>
parents: 3582
diff changeset
59
3582
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
60 def do_run_migrations(connection):
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
61 context.configure(
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
62 connection=connection,
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
63 target_metadata=target_metadata,
3743
54c249ec35ce core (memory/migration): ignore FTS table when autogenerating script for migration:
Goffi <goffi@goffi.org>
parents: 3582
diff changeset
64 render_as_batch=True,
54c249ec35ce core (memory/migration): ignore FTS table when autogenerating script for migration:
Goffi <goffi@goffi.org>
parents: 3582
diff changeset
65 include_name=include_name
3582
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
66 )
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
67
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
68 with context.begin_transaction():
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
69 context.run_migrations()
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
70
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
71
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
72 async def run_migrations_online():
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
73 """Run migrations in 'online' mode.
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
74
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
75 In this scenario we need to create an Engine
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
76 and associate a connection with the context.
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
77
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
78 """
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
79 db_config = sqla_config.getDbConfig()
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
80 engine = create_async_engine(
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
81 db_config["url"],
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
82 poolclass=pool.NullPool,
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
83 future=True,
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
84 )
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
85
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
86 async with engine.connect() as connection:
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
87 await connection.run_sync(do_run_migrations)
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
88
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
89
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
90 if context.is_offline_mode():
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
91 run_migrations_offline()
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
92 else:
71516731d0aa core (memory/sqla): database migration using Alembic:
Goffi <goffi@goffi.org>
parents:
diff changeset
93 asyncio.run(run_migrations_online())