sat_docs: flatpak/build_manifest.py annotate

annotate flatpak/build_manifest.py @ 175:73053fcc74c2

association: ajout du compte rendu de l'AG extraordinaire du 30 octobre 2021

author	Goffi <goffi@goffi.org>
date	Thu, 02 Dec 2021 21:21:36 +0100
parents	a213053a03be
children

rev	line source
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1 #!/usr/bin/env python3
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	2
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	3 import tempfile
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	4 import subprocess
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	5 from pathlib import Path
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	6 from typing import Optional, List
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	7 from dataclasses import dataclass
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	8 import hashlib
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	9 from ftplib import FTP
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	10 from urllib.parse import urlparse
139 d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	11 from textwrap import dedent
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	12 import sys
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	13 import os
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	14 import json
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	15 import time
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	16 import argparse
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	17 import shutil
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	18 from packaging.version import parse as parse_version
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	19 import requests
139 d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	20 from lxml import etree
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	21
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	22
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	23 CACHE_LIMIT = 3600 * 24
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	24 PYTHON_DEP_OVERRIDE = {
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	25 "pygments": {
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	26 "build-commands": [
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	27 # pygments being in freedesktop SDK, we need to use --ignore-installed flag
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	28 'pip3 install --ignore-installed --exists-action=i --no-index --find-links='
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	29 '"file://${PWD}" --prefix=${FLATPAK_DEST} . --no-build-isolation'
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	30 ]
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	31 }
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	32 }
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	33 PYTHON_DEP_PREINSTALL = [
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	34 "wheel",
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	35 ]
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	36 PYTHON_DEP_REQUIREMENTS_UPDATE = {
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	37 # those modules are not seen as a twisted requirement, so they're sometimes misplaced
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	38 'twisted': ['service-identity', 'pyopenssl'],
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	39 }
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	40 PYTHON_SETUP_REQUIREMENTS = [
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	41 # needed for cryptography
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	42 'toml',
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	43 'semantic_version',
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	44
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	45 'setuptools_scm',
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	46 'docutils', # needed by m2r
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	47 'mistune', # needed by m2r
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	48 'm2r', # needed by automat
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	49 ]
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	50 DEFAULT_MANIFEST = {
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	51 "runtime": "org.freedesktop.Platform",
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	52 "runtime-version": "20.08",
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	53 "sdk": "org.freedesktop.Sdk",
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	54 "command": "libervia_wrapper",
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	55 "finish-args": [
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	56 "--socket=session-bus",
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	57 "--share=network",
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	58 "--filesystem=home"
153 df75b62e653e flatpak (build_manifest): some fixes needed after runtime update: Goffi <goffi@goffi.org> parents: 151 diff changeset	59 ],
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	60 }
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	61 SHOW_REQUIRES_HEADER = 'Requires: '
161 4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	62 SHOW_VERSION_HEADER = 'Version: '
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	63 SETTINGS_KEY = '_build_settings'
139 d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	64 APPDATA_RELEASE_DEV_TEXT = dedent("""\
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	65 This is a development version, used as a preview.
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	66 Please note that it is incomplete and it probably contains bugs.
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	67 """)
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	68 OVERWRITE_WARNING = "{} already exists, do you want to overwrite it (y/N)? "
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	69
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	70 # packages already installed in flatpak runtime
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	71 PYTHON_DEPS_IGNORE = ("setuptools", "cython", "six", "markdown")
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	72 PYTHON_VERSION_FORCE = {
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	73 # we have to restrict idna due to requests's idna<3 requirements
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	74 # (as of requests 2.25.1)
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	75 # TODO: check if this requirements is still there
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	76 'idna': '2.10',
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	77 # next versions need RUST
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	78 # TODO: upgrade
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	79 'cryptography': '3.3.2',
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	80 }
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	81
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	82
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	83 PIP_CMD = (
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	84 'pip3 install --exists-action=i --no-index --find-links="file://${{PWD}}" '
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	85 '--prefix=${{FLATPAK_DEST}} {package} --no-build-isolation'
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	86 )
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	87
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	88 pkg_repos = {}
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	89 python_setup_requirements = PYTHON_SETUP_REQUIREMENTS.copy()
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	90
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	91
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	92 @dataclass
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	93 class Package:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	94 name: str
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	95 version: str
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	96 hash_: str
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	97 url: str
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	98 requirements: List[str]
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	99 repos: Optional[dict]
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	100
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	101 def __hash__(self):
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	102 return hash(self.name)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	103
161 4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	104 def __eq__(self, other):
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	105 return self.name == other.name
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	106
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	107
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	108 def print_step(step):
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	109 print(
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	110 "\n"
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	111 f"┌{'─' * (len(step) + 2)}┐\n"
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	112 f"│ {step} │\n"
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	113 f"└{'─' * (len(step) + 2)}┘\n"
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	114 "\n"
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	115 )
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	116
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	117
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	118 def parse_args():
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	119 parser = argparse.ArgumentParser(
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	120 description='Flatpak manifest builder for Libervia')
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	121
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	122 build_group = parser.add_argument_group('building', 'options used to generate the manifest')
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	123 export_group = parser.add_argument_group('export', 'otions used to building files')
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	124
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	125 # build group
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	126 build_group.add_argument('-f', '--force', action="store_true",
139 d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	127 help="force overwritting of existing manifest and appdata file")
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	128 build_group.add_argument('--ignore-cache', action='append', default=[],
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	129 help='ignore the cache of this step ("all" to ignore all caches)')
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	130 build_group.add_argument(
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	131 '--deps-dir',
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	132 help="use this directory to build_group python dependencies (it won't be deleted at "
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	133 "the end, useful when you want to re-use it and avoir re-downloading)")
140 e23e414987d7 flatpak (build_manifest): --no-binary is currently disabled as version parsing doesn't handle yet wheel packages. Goffi <goffi@goffi.org> parents: 139 diff changeset	134 # build_group.add_argument('--no-binary', help="don't use binary packages")
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	135
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	136 # export group
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	137 export_group.add_argument('-s', '--symlink', choices=['no', 'all', 'cache'],
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	138 default='no',
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	139 help='"no" to copy all files, "all" to symlink all files, "cache" to copy files '
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	140 'and symlink only cache (default: "no")')
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	141 export_group.add_argument('-F', '--force-export', action='store_true',
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	142 help='force overwritting of existing files/symlinks when exporting (DEFAULT: '
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	143 'existing files are skipped)')
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	144 export_group.add_argument('-e', '--export', type=Path,
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	145 help="export build files to this directory (DEFAULT: don't export files)")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	146
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	147 # common
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	148 parser.add_argument('name', type=str, help="name of the package to build_group")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	149 parser.add_argument('version', type=str, help="version of the package to build_group")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	150
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	151 args = parser.parse_args()
140 e23e414987d7 flatpak (build_manifest): --no-binary is currently disabled as version parsing doesn't handle yet wheel packages. Goffi <goffi@goffi.org> parents: 139 diff changeset	152 # FIXME: no_binary is currently not managed because version parser doesn't handle
e23e414987d7 flatpak (build_manifest): --no-binary is currently disabled as version parsing doesn't handle yet wheel packages. Goffi <goffi@goffi.org> parents: 139 diff changeset	153 # wheels
e23e414987d7 flatpak (build_manifest): --no-binary is currently disabled as version parsing doesn't handle yet wheel packages. Goffi <goffi@goffi.org> parents: 139 diff changeset	154 args.no_binary = True
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	155
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	156 if 'all' in args.ignore_cache and len(args.ignore_cache) != 1:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	157 parser.error('"all" cannot be used with step names in --ignore-cache, '
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	158 'use it alone')
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	159
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	160 if args.export is not None and not args.export.is_dir():
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	161 parser.error(f"{args.export} is not a directory!")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	162
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	163 return args
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	164
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	165
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	166 ## useful methods ##
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	167
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	168 def get_cache(name):
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	169 """Retrieve cache for a step
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	170
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	171 cache will be ignored if too old, or explicitly ignored by user
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	172 @param name(str): name of the step
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	173 @return (object): cached data
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	174 """
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	175 # name of the cache without prefix, mainly used for python_deps
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	176 shortname = name.split('__', 1)[0]
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	177
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	178 if shortname in args.ignore_cache or 'all' in args.ignore_cache:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	179 print(f"ignoring cache for {shortname}")
154 cd411b0d1e19 flatpak (build_manifest): fixed ignore-cache when "all" is used Goffi <goffi@goffi.org> parents: 153 diff changeset	180 if shortname in args.ignore_cache:
cd411b0d1e19 flatpak (build_manifest): fixed ignore-cache when "all" is used Goffi <goffi@goffi.org> parents: 153 diff changeset	181 args.ignore_cache.remove(shortname)
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	182 return None
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	183 try:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	184 cache_name = cache[name]
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	185 cache_updated = cache_name['updated']
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	186 except KeyError:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	187 return None
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	188 if time.time() - cache_updated > CACHE_LIMIT:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	189 print(f"cache for {name} is too old, we won't use it")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	190 return None
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	191 print(f"cache found for {shortname}")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	192 return cache_name['data']
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	193
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	194
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	195 def set_cache(name, data):
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	196 """Update cache for a step
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	197
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	198 @param name(str): name of the step
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	199 @param data(object): data to cache
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	200 """
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	201 cache_name = cache.setdefault(name, {})
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	202 cache_name['updated'] = time.time()
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	203 cache_name['data'] = data
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	204 with cache_path.open('w') as f:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	205 json.dump(cache, f)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	206
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	207
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	208 def get_python_package(package_name, step_name=None, step_message=None, with_pre=False,
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	209 version=None):
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	210 """Generate module for a Python package
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	211
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	212 @param package_name(str, list[str]): name of the Python package
157 150085e39072 flatpak (build_manifest): version can be forced in setup_requirements + force cython version in cagou template Goffi <goffi@goffi.org> parents: 156 diff changeset	213 use list of str to get several packages at once.
150085e39072 flatpak (build_manifest): version can be forced in setup_requirements + force cython version in cagou template Goffi <goffi@goffi.org> parents: 156 diff changeset	214 A (name, version) tuple (or any size 2 iterable) can be used to force version of a
150085e39072 flatpak (build_manifest): version can be forced in setup_requirements + force cython version in cagou template Goffi <goffi@goffi.org> parents: 156 diff changeset	215 package.
150085e39072 flatpak (build_manifest): version can be forced in setup_requirements + force cython version in cagou template Goffi <goffi@goffi.org> parents: 156 diff changeset	216 If package is a list with iterables, it will be modified in place to keep only the
150085e39072 flatpak (build_manifest): version can be forced in setup_requirements + force cython version in cagou template Goffi <goffi@goffi.org> parents: 156 diff changeset	217 name as string. This is because version is not needed anymore after that, but
150085e39072 flatpak (build_manifest): version can be forced in setup_requirements + force cython version in cagou template Goffi <goffi@goffi.org> parents: 156 diff changeset	218 name is still used to remove dependencies already installed with setup
150085e39072 flatpak (build_manifest): version can be forced in setup_requirements + force cython version in cagou template Goffi <goffi@goffi.org> parents: 156 diff changeset	219 requirements.
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	220 @param step_name(None, str): name of the step, None to use package_name
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	221 @param step_message(None, str): message of the step (None to use default one)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	222 use empty string to disable it
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	223 @param with_pre(bool): if True retrieve also pre-releases
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	224 @param version(str, None): indicate the version of the package to download
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	225 if None, the most recent version compativle with `with_pre` will be used
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	226 """
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	227 single = isinstance(package_name, str)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	228 if step_name is None:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	229 step_name = package_name if single else ' - '.join(package_name)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	230 if step_message != "":
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	231 print_step(step_message or f"retrieving latest version of {package_name}")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	232 cache = get_cache(step_name)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	233 if cache is not None:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	234 return cache
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	235 package_names = [package_name] if single else package_name
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	236 data = []
157 150085e39072 flatpak (build_manifest): version can be forced in setup_requirements + force cython version in cagou template Goffi <goffi@goffi.org> parents: 156 diff changeset	237
150085e39072 flatpak (build_manifest): version can be forced in setup_requirements + force cython version in cagou template Goffi <goffi@goffi.org> parents: 156 diff changeset	238 for idx, name in enumerate(package_names):
150085e39072 flatpak (build_manifest): version can be forced in setup_requirements + force cython version in cagou template Goffi <goffi@goffi.org> parents: 156 diff changeset	239 if not isinstance(name, str):
150085e39072 flatpak (build_manifest): version can be forced in setup_requirements + force cython version in cagou template Goffi <goffi@goffi.org> parents: 156 diff changeset	240 if version is not None:
150085e39072 flatpak (build_manifest): version can be forced in setup_requirements + force cython version in cagou template Goffi <goffi@goffi.org> parents: 156 diff changeset	241 raise ValueError(
150085e39072 flatpak (build_manifest): version can be forced in setup_requirements + force cython version in cagou template Goffi <goffi@goffi.org> parents: 156 diff changeset	242 "forced version can't be used with packages versions (i.e. tuples)")
150085e39072 flatpak (build_manifest): version can be forced in setup_requirements + force cython version in cagou template Goffi <goffi@goffi.org> parents: 156 diff changeset	243 name, forced_version = name
150085e39072 flatpak (build_manifest): version can be forced in setup_requirements + force cython version in cagou template Goffi <goffi@goffi.org> parents: 156 diff changeset	244 package_names[idx] = name
150085e39072 flatpak (build_manifest): version can be forced in setup_requirements + force cython version in cagou template Goffi <goffi@goffi.org> parents: 156 diff changeset	245 else:
150085e39072 flatpak (build_manifest): version can be forced in setup_requirements + force cython version in cagou template Goffi <goffi@goffi.org> parents: 156 diff changeset	246 forced_version = version
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	247 r = requests.get(f"https://pypi.org/pypi/{name}/json")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	248 r.raise_for_status()
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	249
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	250 releases_data = []
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	251 for version_raw, release_data in r.json()["releases"].items():
157 150085e39072 flatpak (build_manifest): version can be forced in setup_requirements + force cython version in cagou template Goffi <goffi@goffi.org> parents: 156 diff changeset	252 if forced_version is not None and forced_version != version_raw:
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	253 continue
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	254 release_version = parse_version(version_raw)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	255 if release_version.is_prerelease and not with_pre:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	256 continue
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	257 releases_data.append((release_version, release_data))
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	258
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	259 # we sort releases by version to be sure to have latests one
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	260 releases_data.sort(key=lambda d: d[0])
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	261
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	262 try:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	263 release_version, release_data = releases_data[-1]
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	264 package = next(r for r in release_data if r["packagetype"] == "sdist")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	265 except (IndexError, StopIteration):
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	266 raise RuntimeError(f"Can't find a matching package for {name}")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	267
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	268 print(f"{name} {release_version} will be used")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	269 dep_data = {
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	270 "name": name,
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	271 "buildsystem": "simple",
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	272 "build-commands": [
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	273 PIP_CMD.format(package=".")
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	274 ],
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	275 "sources": [
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	276 {
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	277 "type": "archive",
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	278 "url": package["url"],
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	279 "sha256": package["digests"]["sha256"],
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	280 }
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	281 ],
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	282 }
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	283 data.append(dep_data)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	284
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	285 set_cache(step_name, data)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	286 return data
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	287
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	288
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	289 def file_upload(filename, dest="/app/bin", src=None, replace=None, use_install=False):
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	290 """Generate manifest modules to upload a local file
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	291
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	292 @param filename(str): name of the local file to upload
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	293 Note that it will be overwritted if replace is used
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	294 @param dest(str, Path): path where the file must be installed in the container
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	295 @param src(str, None): when replace is used, name of the source file
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	296 @param replace(dict, None): mapping of text to replace
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	297 Use when file must be dynamicly adapted to the container, note that src must be
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	298 set to the name of the template file if this is used.
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	299 e.g. {'##COMMAND##', 'cagou'} will replace all '##COMMAND##' in filename
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	300 by 'cagou'
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	301 @param use_install(bool): if True, install file with `install src dest`
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	302 else, use `mv src dest`
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	303 @return (list(dict)): modules to add to the manifest
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	304 """
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	305 assert isinstance(filename, str) and '/' not in filename
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	306 print_step(f"generating {filename} module")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	307 # no cache for this one, to be sure to have always latest version
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	308 filename = Path(filename)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	309 if src is None:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	310 file_to_test = filename
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	311 else:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	312 src = Path(src)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	313 file_to_test = src
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	314 if not file_to_test.exists():
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	315 raise RuntimeError(
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	316 f"{file_to_test} is missing, it is needed to build the manifest!")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	317
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	318 if replace:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	319 if src is None:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	320 raise ValueError(
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	321 '"src" must be set to name of template file if replace is used')
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	322 print(f'doing replacement in template "{src}" to dest "{filename}"')
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	323 # there are strings to replace, we read file, do the changes and dump the
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	324 # result in <filename>
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	325 with open(src) as f:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	326 buff = f.read()
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	327
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	328 for old, new in replace.items():
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	329 buff = buff.replace(old, new)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	330
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	331 with open(filename, 'w') as f:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	332 f.write(buff)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	333 else:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	334 if src is not None:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	335 raise ValueError('"src" must not be used if replace is not used')
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	336
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	337 with filename.open('rb') as f:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	338 hash_ = hashlib.sha256(f.read()).hexdigest()
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	339
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	340 dest = Path(dest)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	341
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	342 dep_data = {
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	343 "name": str(filename),
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	344 "buildsystem": "simple",
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	345 "build-commands": [
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	346 ],
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	347 "sources": [
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	348 {
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	349 "type": "file",
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	350 "path": str(filename),
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	351 "sha256": hash_,
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	352 }
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	353 ]
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	354 }
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	355
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	356 build_commands = dep_data['build-commands']
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	357 if use_install:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	358 build_commands.append(f"install -Dv {filename} {dest}")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	359 else:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	360 if dest.as_posix() not in ('/app', '/app/bin'):
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	361 # just in case the destination directory doesn't exist
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	362 build_commands.append(f"mkdir -p {dest.parent}")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	363 build_commands.append(f"mv -v {filename} {dest}")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	364
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	365 return [dep_data]
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	366
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	367
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	368 def file_from_url(url, dest=None, step_name=None, step_message=None):
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	369 """Generate manifest modules for a file either on a http(s) url or local
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	370
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	371 @param url(str): url of the file to use, or local path
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	372 if it starts with http, url will be used, else file_upload(url) will be used
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	373 @param dest(str, None): path were the file should be copied
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	374 """
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	375 parsed = urlparse(url)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	376 if not parsed.scheme:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	377 return file_upload(url)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	378 if not parsed.scheme.startswith('http'):
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	379 raise NotImplementedError (
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	380 f'we can\'t use the URL "{url}", the scheme "{parsed.scheme}" is not managed')
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	381
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	382 filepath = Path(parsed.path)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	383 stem = filepath.stem
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	384
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	385 if step_name is None:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	386 # we use this name to easily ignore cache (with stem) while avoiding
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	387 # conflict if we have 2 URLs with the same stem
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	388 step_name = f"{stem}__{url}"
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	389
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	390 if step_message is None:
139 d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	391 step_message = f"generating module for {stem}"
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	392
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	393 print_step(step_message)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	394 cache = get_cache(step_name)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	395 if cache is not None:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	396 return cache
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	397
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	398 r = requests.get(url)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	399 r.raise_for_status()
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	400 file_hash = hashlib.sha256(r.content).hexdigest()
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	401
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	402 dep_data = {"name": stem}
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	403
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	404 if dest is not None:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	405 dest = Path(dest)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	406 dep_data.update({
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	407 'buildsystem': 'simple',
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	408 'build-commands':[
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	409 f'mkdir -p {dest.parent}',
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	410 f'mv "{filepath.name}" "{dest}"',
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	411 ]
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	412 })
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	413
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	414 dep_data['sources'] = [
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	415 {
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	416 "type": "file",
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	417 "url": url,
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	418 "sha256": file_hash,
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	419 }
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	420 ]
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	421
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	422 data = [dep_data]
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	423 set_cache(step_name, data)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	424 return data
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	425
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	426
161 4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	427 def get_requirements(pip, package_name, done=None, requirements=None, tree=None, indent=0):
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	428 """Retrieve requirements for an installed python package
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	429
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	430 @param pip(Path): path to pip executable to use
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	431 package must be installed in the environment of this pip
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	432 @param package_name(str): name of the package to retrieve
161 4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	433 @return(list[(str,str)]): list of ordered requirements (package_name, version)
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	434 """
161 4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	435 print(f"found dependency: {package_name}")
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	436 if requirements is None:
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	437 assert done is None
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	438 assert tree is None
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	439 if not package_name == canonical(package_name):
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	440 raise ValueError("main package name is not canonical")
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	441 tree = []
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	442 requirements = []
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	443 done = []
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	444 main = True
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	445 else:
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	446 main = False
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	447 done.append(package_name)
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	448 show_cplted = subprocess.run([pip, "show", package_name],
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	449 capture_output=True, text=True)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	450 show_cplted.check_returncode()
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	451
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	452 lines = show_cplted.stdout.split('\n')
161 4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	453 version_raw = next(l for l in lines if l.startswith(SHOW_VERSION_HEADER))
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	454 version = version_raw[len(SHOW_VERSION_HEADER):]
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	455 requirement_raw = next(l for l in lines if l.startswith(SHOW_REQUIRES_HEADER))
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	456 requirement_raw = requirement_raw[len(SHOW_REQUIRES_HEADER):]
161 4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	457 requirements_canon = [canonical(p) for p in requirement_raw.split(',') if p.strip()]
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	458 requirements_update = [
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	459 canonical(r) for r in PYTHON_DEP_REQUIREMENTS_UPDATE.get(package_name, [])]
161 4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	460 new_requirements = set(requirements_update).difference(requirements_canon)
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	461 if new_requirements:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	462 print("adding extra requirements to {}: {}".format(
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	463 package_name, ', '.join(new_requirements)))
161 4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	464 requirements_canon.extend(new_requirements)
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	465 tree.append(f"{' '*indent}{package_name} ({version})")
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	466 for r in requirements_canon:
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	467 if r not in done:
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	468 get_requirements(pip, r, done, requirements, tree, indent+1)
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	469 req_tuple = (package_name, version, requirements_canon)
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	470 requirements.append(req_tuple)
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	471 if main:
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	472 print("\ndependency tree:\n")
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	473 print('\n'.join(tree))
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	474 return requirements
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	475
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	476
139 d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	477 def get_hg_id_date(path):
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	478 """Get short identifier and date of current commit from given Mercurial repository
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	479
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	480 version is retrieve with `hg id`, a "+" is appended after shortrev if it has
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	481 been modified.
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	482 @param path(str, Path): path to the repository
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	483 @return(tuple(str, date)): found revision + iso date
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	484 """
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	485 hg_cplted = subprocess.run(
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	486 ["hg", "id", "--template", "{id\|short}{dirty}\n{date\|isodate}", path],
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	487 capture_output=True, text=True)
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	488 hg_cplted.check_returncode()
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	489 return hg_cplted.stdout.split('\n')
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	490
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	491
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	492 def get_cache_dir():
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	493 """Return path to directory to use for cache"""
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	494 return Path(f"cache_{app_id}")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	495
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	496
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	497 def canonical(name):
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	498 """Get canonical name of a package"""
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	499 return name.lower().replace('_', '-').strip()
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	500
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	501
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	502 modules = []
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	503 cache_path = Path.home() / Path('.cache/sat_flatpak_cache.json')
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	504 if not cache_path.exists():
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	505 cache = {}
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	506 else:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	507 with cache_path.open() as f:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	508 cache = json.load(f)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	509
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	510
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	511 ## steps ##
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	512
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	513 def get_libxslt():
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	514 """Generate manifest module to install libxslt (needed for lxml)"""
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	515 step_name = 'libxslt'
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	516 print_step("retrieving latest version of libxslt")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	517 cache = get_cache(step_name)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	518 if cache is not None:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	519 return cache
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	520
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	521 ftp = FTP("xmlsoft.org")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	522 ftp.login()
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	523 ftp.cwd("libxml2")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	524 libxslt_archives = [l for l in ftp.nlst() if 'xslt' in l
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	525 and l.endswith('tar.gz')
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	526 and 'git' not in l
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	527 and 'rc' not in l]
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	528 latest_libxslt = libxslt_archives[-1]
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	529 print(f"latest libxslt found: {latest_libxslt}")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	530
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	531 with tempfile.TemporaryDirectory() as tmp_dirname:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	532 tmp_dir = Path(tmp_dirname)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	533 file_path = tmp_dir / latest_libxslt
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	534 with file_path.open('wb+') as f:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	535 ftp.retrbinary('RETR ' + latest_libxslt, f.write)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	536 f.seek(0)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	537 libxslt_hash = hashlib.sha256(f.read()).hexdigest()
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	538
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	539 ftp.quit()
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	540
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	541 print(f"libxstl hash: {libxslt_hash}")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	542
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	543 data = [{
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	544 "name": "libxslt",
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	545 "sources": [
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	546 {
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	547 "type": "archive",
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	548 "url": f"ftp://xmlsoft.org/libxml2/{latest_libxslt}",
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	549 "sha256": libxslt_hash,
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	550 }
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	551 ]
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	552 }]
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	553 set_cache(step_name, data)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	554 return data
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	555
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	556
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	557 def get_python_deps():
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	558 """Generate manifest modules for python dependencies of main package"""
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	559 step_name = f'python_deps__{app_id}'
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	560 print_step("retrieving python dependencies")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	561 cache = get_cache(step_name)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	562 if cache is not None:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	563 return cache
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	564
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	565 with tempfile.TemporaryDirectory() as tmp_dirname:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	566 if args.deps_dir is not None:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	567 # we ignore the created temporary directory is we have an other one specified
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	568 tmp_dirname = args.deps_dir
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	569 tmp_dir = Path(tmp_dirname)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	570 env_dir = tmp_dir / 'env'
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	571 pip = env_dir / 'bin' / 'pip'
161 4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	572 if env_dir.exists():
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	573 print("packages are already installed")
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	574 else:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	575 print(f"working in temporary directory {tmp_dirname}")
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	576 venv_cplted = subprocess.run(["/usr/bin/env", "python3", "-m", "venv", env_dir])
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	577 venv_cplted.check_returncode()
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	578 print("\ninstalling package\n")
161 4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	579 if PYTHON_DEP_PREINSTALL:
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	580 print("preinstalling packages")
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	581 for dep in PYTHON_DEP_PREINSTALL:
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	582 inst_cplted = subprocess.run([pip, "install", dep])
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	583 inst_cplted.check_returncode()
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	584
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	585 # we install package to have a good map of requirements, pypi metadata
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	586 # are incomplete. Packages should be retrieved from pip cache
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	587 if isinstance(main_package_source, Path):
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	588 inst_cplted = subprocess.run(
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	589 [pip, "install", "--progress-bar", "emoji", "-r", "requirements.txt"],
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	590 cwd = main_package_source
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	591 )
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	592 else:
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	593 inst_cplted = subprocess.run(
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	594 [pip, "install", "--progress-bar", "emoji", main_package_source])
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	595 inst_cplted.check_returncode()
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	596
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	597
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	598 print("checking package coming from VCS")
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	599 freeze_cplted = subprocess.run(
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	600 [pip, "freeze"], capture_output=True, text=True
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	601 )
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	602 freeze_cplted.check_returncode()
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	603 lines = freeze_cplted.stdout.split('\n')
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	604 for line in lines:
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	605 if " @ " in line:
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	606 pkg, reference = line.split(" @ ")
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	607 pkg = canonical(pkg)
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	608 if pkg == main_package:
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	609 continue
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	610 if reference.startswith("hg+http"):
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	611 url = reference[3:].split('@')[0]
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	612 pkg_repos[pkg] = {"type": "hg", "url": url}
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	613 print(f"found {pkg!r} from {url!r} (Mercurial)")
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	614
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	615 print_step("analyzing python dependencies")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	616 deps_map = {}
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	617
161 4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	618 requirements = get_requirements(pip, main_package)
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	619
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	620
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	621 print("\nretrieveing metadata on PyPi\n")
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	622
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	623 if not args.no_binary:
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	624 print(r"/!\ retrieving binary packages is not currently supported")
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	625
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	626 deps = []
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	627 # deps coming from repository, we put at the end to make rebuildind quicker
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	628 repos_deps = []
161 4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	629 for name, version, pkg_requirements in requirements:
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	630 print(f"{name}: ", end='')
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	631 if name in PYTHON_DEPS_IGNORE:
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	632 print(f"{name!r} ignored")
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	633 continue
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	634 if name in pkg_repos:
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	635 # main_package must be in normal dependencies
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	636 target_deps = repos_deps if name != main_package else deps
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	637 repos = pkg_repos[name]
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	638 dep_hash = ""
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	639 url = ""
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	640 source_release = {"packagetype": f"VCS ({repos['type']})"}
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	641 else:
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	642 target_deps = deps
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	643 r = requests.get(f"https://pypi.org/pypi/{name}/{version}/json")
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	644 r.raise_for_status()
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	645 dep_json = r.json()
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	646 release_json = dep_json["releases"][version]
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	647 source_release = next(
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	648 r for r in release_json if r['packagetype'] == 'sdist'
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	649 )
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	650 dep_hash = source_release['digests']['sha256']
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	651 url = source_release['url']
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	652 repos = None
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	653 dep = Package(
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	654 name=name,
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	655 version=version,
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	656 hash_=dep_hash,
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	657 url=url,
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	658 requirements=pkg_requirements,
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	659 repos=repos,
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	660 )
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	661 target_deps.append(dep)
161 4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	662 deps_map[name] = dep
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	663 print(f"version: {version}, type: {source_release['packagetype']}")
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	664
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	665 # we remove deps already installed with PYTHON_SETUP_REQUIREMENTS
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	666 for name in python_setup_requirements:
161 4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	667 if not isinstance(name, str):
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	668 # this is a package with version in a list, first item is the name
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	669 name = name[0]
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	670 package_data = deps_map.get(name)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	671 if package_data is not None:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	672 deps.remove(package_data)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	673 print(f"removed {name} which is already a setup requirement")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	674
161 4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	675 # main package is installed at the end of the manifest
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	676 # so we remove it (it is the last package in deps)
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	677 main_pack = deps.pop()
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	678 assert main_pack.name == main_package
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	679
161 4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	680 print("\nwe now generate modules for python dependencies\n")
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	681 data = []
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	682 version_force = {
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	683 canonical(n): v
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	684 for n,v in build_settings.get('python_deps_version_force', {}).items()
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	685 }
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	686
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	687 for dep in deps + repos_deps:
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	688 version_forced = version_force.get(dep.name)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	689 if version_forced == 'pre-release':
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	690 print(f"using pre-release version for {dep.name} as requested in build "
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	691 "settings")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	692 dep_data = get_python_package(dep.name, step_message='', with_pre=True)[0]
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	693 elif version_forced is not None:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	694 print(f"using version {version_forced} for {dep.name} as requested in "
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	695 "build settings")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	696 dep_data = get_python_package(dep.name, step_message='',
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	697 version=version_forced)[0]
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	698 elif dep.repos is not None:
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	699 package_source = cache_from_repos(dep.name, dep.repos)
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	700 dep_data = get_repos_module(dep.name, package_source)
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	701 else:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	702 dep_data = {
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	703 "name": dep.name,
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	704 "buildsystem": "simple",
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	705 "build-commands": [
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	706 PIP_CMD.format(package=".")
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	707 ],
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	708 "sources": [
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	709 {
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	710 "type": "archive",
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	711 "url": dep.url,
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	712 "sha256": dep.hash_,
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	713 }
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	714 ],
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	715 }
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	716
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	717 if dep.name in PYTHON_DEP_OVERRIDE:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	718 print(f"applying override for {dep.name}")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	719 override = PYTHON_DEP_OVERRIDE[dep.name]
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	720
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	721 # key suffixed with "_extend" won't replace the data
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	722 for key in list(override.keys()):
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	723 if key.endswith('_extend'):
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	724 real_key = key[:-7]
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	725 extend_data = override.pop(key)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	726 if real_key == 'sources':
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	727 for item_data in extend_data:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	728 # we look for missing hashes and add them
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	729 if (item_data.get('type') in ('file', 'patch')
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	730 and 'sha256' not in item_data):
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	731 with open(item_data['path'], 'rb') as f:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	732 hash_ = hashlib.sha256(f.read()).hexdigest()
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	733 item_data['sha256'] = hash_
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	734
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	735 dep_data.setdefault(real_key, []).extend(extend_data)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	736
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	737 dep_data.update(override)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	738
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	739 data.append(dep_data)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	740
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	741 set_cache(step_name, data)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	742 return data
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	743
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	744
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	745 def cache_from_repos(name, dev_repos):
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	746 """Get and cache locally a repository and returns cache path"""
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	747 print(f"retrieving code from repository {dev_repos!r}")
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	748 repos_type = dev_repos.get('type', 'hg')
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	749 if repos_type != 'hg':
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	750 raise NotImplementedError("only Mercurial is managed so far")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	751 url = dev_repos['url']
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	752 dest_path = get_cache_dir() / name
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	753 dest_path.mkdir(parents=True, exist_ok=True)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	754 if (dest_path / '.hg').is_dir():
156 0a92bb60db12 flatpak (build_manifest): Mercurial repository is now updated if it is already cloned Goffi <goffi@goffi.org> parents: 155 diff changeset	755 print("code is already in cache, updating")
0a92bb60db12 flatpak (build_manifest): Mercurial repository is now updated if it is already cloned Goffi <goffi@goffi.org> parents: 155 diff changeset	756 hg_cplted = subprocess.run(["hg", "pull", "-u", "--cwd", dest_path])
0a92bb60db12 flatpak (build_manifest): Mercurial repository is now updated if it is already cloned Goffi <goffi@goffi.org> parents: 155 diff changeset	757 hg_cplted.check_returncode()
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	758 else:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	759 try:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	760 print(f"retrieving code from repository {url} into {dest_path}")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	761 hg_cplted = subprocess.run(["hg", "clone", url, dest_path])
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	762 hg_cplted.check_returncode()
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	763 except Exception as e:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	764 shutil.rmtree(dest_path)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	765 raise e
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	766 return dest_path
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	767
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	768
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	769 def get_repos_module(package, source):
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	770 """Generate manifest module for a locally downloaded repository"""
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	771 dep_data = {
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	772 "name": package,
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	773 "buildsystem": "simple",
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	774 "build-commands": [
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	775 PIP_CMD.format(package=".")
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	776 ],
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	777 "sources": [
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	778 {
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	779 "type": "dir",
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	780 "path": str(source)
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	781 }
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	782 ],
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	783 }
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	784 return dep_data
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	785
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	786
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	787 def get_sat_media():
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	788 """Generate module for last version of sat_media available on the FTP"""
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	789 step_name = 'sat_media'
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	790 print_step("retrieving latest version of sat_media")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	791 cache = get_cache(step_name)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	792 if cache is not None:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	793 return cache
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	794 url = "https://ftp.goffi.org/sat_media/sat_media.tar.bz2"
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	795 r = requests.get(url)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	796 r.raise_for_status()
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	797 hash_ = hashlib.sha256(r.content).hexdigest()
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	798 dep_data = {
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	799 "name": "sat-media",
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	800 "buildsystem": "simple",
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	801 "build-commands": [
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	802 "cp -vr . ${FLATPAK_DEST}/share/sat-media"
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	803 ],
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	804 "sources": [
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	805 {
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	806 "type": "archive",
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	807 "url": url,
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	808 "sha256": hash_,
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	809 }
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	810 ],
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	811 }
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	812
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	813 data = [dep_data]
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	814 set_cache(step_name, data)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	815 return data
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	816
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	817
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	818 def get_icon():
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	819 icon = build_settings.get('icon')
145 9201caf7b2a8 flatpak (build_manifest): don't crash if icon is not specified Goffi <goffi@goffi.org> parents: 140 diff changeset	820 if icon is None:
9201caf7b2a8 flatpak (build_manifest): don't crash if icon is not specified Goffi <goffi@goffi.org> parents: 140 diff changeset	821 return []
9201caf7b2a8 flatpak (build_manifest): don't crash if icon is not specified Goffi <goffi@goffi.org> parents: 140 diff changeset	822 else:
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	823 if isinstance(icon, str):
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	824 icon = {'url': icon}
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	825 icon_path = Path(urlparse(icon['url']).path)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	826 suffix = icon_path.suffix[1:]
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	827 if suffix not in ('svg', 'png'):
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	828 raise ValueError("invalid icon, you must use a SVG or PNG image!")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	829 if 'size' not in icon:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	830 if suffix == 'svg':
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	831 icon['size'] = 'scalable'
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	832 else:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	833 raise ValueError('icon size is not specified, please add a "size" key')
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	834
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	835 dest_path = f"/app/share/icons/hicolor/{icon['size']}/apps/{app_id}.{suffix}"
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	836
145 9201caf7b2a8 flatpak (build_manifest): don't crash if icon is not specified Goffi <goffi@goffi.org> parents: 140 diff changeset	837 data = file_from_url(
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	838 url = icon['url'],
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	839 dest = dest_path,
150 10adfa3f9552 flatpak (build_manifest): icon names depends of app_id, so set cache to be dependant of app_id Goffi <goffi@goffi.org> parents: 146 diff changeset	840 step_name = f"icon__{app_id}",
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	841 step_message = "retrieving application icon",
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	842 )
145 9201caf7b2a8 flatpak (build_manifest): don't crash if icon is not specified Goffi <goffi@goffi.org> parents: 140 diff changeset	843 data[0]['name'] = 'icon'
9201caf7b2a8 flatpak (build_manifest): don't crash if icon is not specified Goffi <goffi@goffi.org> parents: 140 diff changeset	844 return data
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	845
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	846
139 d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	847 def generate_appdata_from_template(template_file):
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	848 appdata_file = Path(f"{app_id}.appdata.xml")
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	849 if appdata_file.exists() and not args.force:
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	850 confirm = input(OVERWRITE_WARNING.format(appdata_file))
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	851 if confirm != 'y':
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	852 print("manifest building cancelled")
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	853 sys.exit(0)
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	854 parser = etree.XMLParser(remove_blank_text=True)
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	855 tree = etree.parse(template_file, parser)
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	856 root = tree.getroot()
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	857 if args.version == 'dev':
140 e23e414987d7 flatpak (build_manifest): --no-binary is currently disabled as version parsing doesn't handle yet wheel packages. Goffi <goffi@goffi.org> parents: 139 diff changeset	858 print(f"addind release data for dev version ({dev_version_rev})")
139 d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	859 releases_elt = root.find('releases')
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	860 if releases_elt is None:
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	861 raise ValueError(
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	862 "<releases/> element is missing in appdata template, please add it")
161 4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	863 release_elt = etree.Element(
139 d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	864 "release",
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	865 {'type': 'development',
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	866 'version': dev_version_rev,
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	867 'date': dev_version_date},
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	868 )
161 4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	869 releases_elt.insert(0, release_elt)
139 d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	870 description_elt = etree.SubElement(release_elt, 'description')
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	871 text_lines = APPDATA_RELEASE_DEV_TEXT.strip().split('\n')
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	872 for text in text_lines:
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	873 etree.SubElement(description_elt, 'p').text = text
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	874
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	875 with open(appdata_file, "wb") as f:
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	876 f.write(etree.tostring(root, encoding="utf-8", xml_declaration=True,
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	877 pretty_print=True))
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	878
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	879 return appdata_file.as_posix()
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	880
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	881
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	882 def get_app_metadata():
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	883 desktop_file = build_settings.get('desktop_file')
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	884 appdata_file = build_settings.get('appdata_file')
139 d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	885 if desktop_file is None and appdata_file is None:
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	886 return
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	887
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	888 print_step("retrieving application metadata")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	889 # we don't use cache here to be sure to have always up-to-date files
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	890
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	891 data = []
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	892
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	893 if desktop_file is not None:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	894 print("generating module for desktop metadata")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	895 data.extend(file_upload(
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	896 filename = desktop_file,
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	897 dest=f"/app/share/applications/{app_id}.desktop",
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	898 ))
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	899
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	900 if appdata_file is not None:
139 d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	901 if appdata_file.startswith('_tpl_'):
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	902 print("found appdata template, we now use it to generate the file")
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	903 appdata_file = generate_appdata_from_template(appdata_file)
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	904 print("generating module for appdata metadata")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	905 data.extend(file_upload(
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	906 filename = appdata_file,
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	907 dest=f"/app/share/metainfo/{app_id}.appdata.xml",
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	908 ))
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	909
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	910 return data
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	911
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	912
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	913 ## main_script ##
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	914
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	915
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	916 if __name__ == '__main__':
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	917
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	918 args = parse_args()
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	919 app_id = f"org.libervia.{args.name}"
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	920 package_file = Path(f"{app_id}.json")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	921
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	922 print(f"generating manifest for {app_id} ({args.version})")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	923
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	924 if package_file.exists() and not args.force:
139 d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	925 confirm = input(OVERWRITE_WARNING.format(package_file))
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	926 if confirm != 'y':
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	927 print("manifest building cancelled")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	928 sys.exit(0)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	929
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	930 tpl_file = Path(f"_tpl_{package_file}")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	931
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	932 if not tpl_file.exists():
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	933 raise RuntimeError(f"Can't find template {tpl_file}, it is mandatory to build"
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	934 "the manifest!")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	935
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	936 with tpl_file.open() as f:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	937 template = json.load(f)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	938
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	939 build_settings = template.pop(SETTINGS_KEY, {})
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	940 if PYTHON_VERSION_FORCE:
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	941 build_settings.setdefault('python_deps_version_force', {}).update(PYTHON_VERSION_FORCE)
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	942 if "setup_requirements" in build_settings:
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	943 python_setup_requirements.extend(build_settings["setup_requirements"])
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	944 main_package = canonical(build_settings['package'])
139 d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	945 if args.version == 'dev':
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	946 # mercurial is needed for dev version to install but also to
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	947 # retrieve revision used
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	948 python_setup_requirements.append("mercurial")
139 d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	949 if 'dev_repos' not in build_settings:
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	950 raise NotImplementedError(
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	951 "dev version can currently only be handled with a dev repostiory "
d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	952 "(dev_repos)")
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	953 dev_repos = build_settings['dev_repos']
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	954 pkg_repos[main_package] = dev_repos
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	955 print_step("retrieving code from repository")
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	956 main_package_source = cache_from_repos(main_package, dev_repos)
139 d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	957 dev_version_rev, dev_version_date = get_hg_id_date(main_package_source)
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	958 else:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	959 main_package_source = main_package
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	960
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	961 manifest = {}
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	962 manifest['app-id'] = app_id
161 4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	963 if args.version == 'dev':
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	964 manifest['default-branch'] = args.version
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	965 # we update DEFAULT_MANIFEST only now to have "app-id" and "default-branch" on the top
153 df75b62e653e flatpak (build_manifest): some fixes needed after runtime update: Goffi <goffi@goffi.org> parents: 151 diff changeset	966 # of the manifest, also we don't want modules to be set now
df75b62e653e flatpak (build_manifest): some fixes needed after runtime update: Goffi <goffi@goffi.org> parents: 151 diff changeset	967 default_modules = DEFAULT_MANIFEST.pop('modules', [])
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	968 manifest.update(DEFAULT_MANIFEST)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	969 manifest.update(template)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	970
161 4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	971 # modules.extend(get_libxslt())
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	972
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	973 # setup requirements
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	974 modules.extend(get_python_package(
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	975 python_setup_requirements,
146 64c02902b86a flatpak (build_manifest): fixed setup_requirements step cache by making dependent of app_id Goffi <goffi@goffi.org> parents: 145 diff changeset	976 step_name=f"setup_requirements__{app_id}",
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	977 step_message="generating modules for setup requirements")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	978 )
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	979
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	980 # python dependencies
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	981 modules.extend(get_python_deps())
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	982
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	983 # at this point we add things specified in the template
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	984 modules.extend(manifest.get('modules', []))
153 df75b62e653e flatpak (build_manifest): some fixes needed after runtime update: Goffi <goffi@goffi.org> parents: 151 diff changeset	985 modules = default_modules + modules
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	986 manifest['modules'] = modules
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	987
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	988 # sat common things
161 4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	989 existing_modules = {d['name'] for d in modules if not isinstance(d, str)}
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	990 if "sat_templates" not in existing_modules:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	991 modules.extend(get_python_package("sat_templates", with_pre=True))
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	992 modules.extend(get_sat_media())
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	993 conf_file = build_settings.get('conf_file')
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	994 if conf_file:
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	995 modules.extend(file_upload('libervia.conf', '/app', src=conf_file,
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	996 replace={'##DBUS_PREFIX##': app_id},
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	997 use_install=True))
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	998 else:
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	999 modules.extend(file_upload('libervia.conf', '/app'))
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1000
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1001 # wrapper to launch the backend if needed
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	1002 wrapped_command = build_settings['wrapped_command']
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	1003 manifest['command'] = command = f'{wrapped_command}_wrapper'
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	1004 wrapper_template = build_settings.get("wrapper_template", "libervia_wrapper.py")
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	1005 modules.extend(file_upload(command, src=wrapper_template,
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	1006 replace={'##COMMAND##': wrapped_command},
a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	1007 use_install=True))
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1008
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1009 # icon
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1010 modules.extend(get_icon())
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1011
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1012 # desktop file and appdata file
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1013 modules.extend(get_app_metadata())
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1014
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1015 # now the app itself
139 d36a68e396d5 flatpak: set appdata release version for dev version: Goffi <goffi@goffi.org> parents: 138 diff changeset	1016 if args.version == 'dev':
171 a213053a03be flatpak: update files following names change + Python 3 update: Goffi <goffi@goffi.org> parents: 161 diff changeset	1017 modules.append(get_repos_module(main_package, main_package_source))
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1018 else:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1019 modules.extend(get_python_package(main_package, version=args.version))
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1020
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1021 print_step("writing manifest")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1022 with package_file.open('w') as f:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1023 json.dump(manifest, f, indent=4)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1024
154 cd411b0d1e19 flatpak (build_manifest): fixed ignore-cache when "all" is used Goffi <goffi@goffi.org> parents: 153 diff changeset	1025 if args.ignore_cache and args.ignore_cache != ['all']:
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1026 print("/!\\ those --ignore-cache arguments don't correspond to any step: {}".
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1027 format(', '.join(args.ignore_cache)))
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1028
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1029 print(f"manifest generated successfully at {package_file}")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1030
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1031 if args.export is not None:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1032 print_step("exporting building files")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1033 print(f"exporting to {args.export}\n")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1034 to_export = [
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1035 ("package file", package_file),
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1036 ]
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1037
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1038 for m in modules:
161 4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	1039 if isinstance(m, str):
4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	1040 continue
138 274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1041 for s in m.get('sources', []):
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1042 s_type = s.get('type')
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1043 if s_type in ('path', 'file'):
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1044 try:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1045 path = s['path']
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1046 except KeyError:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1047 if 'url' in s:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1048 continue
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1049 else:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1050 raise ValueError(f"path missing for module source:\n{s}")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1051 to_export.append((s_type, Path(path)))
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1052
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1053 for label, path in to_export:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1054 print(f"exporting {label}: {path}")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1055 dest = args.export / path
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1056 if dest.exists():
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1057 if args.force_export:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1058 print(f" replacing existing {dest}")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1059 if path.is_dir():
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1060 shutil.rmtree(dest)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1061 else:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1062 dest.unlink()
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1063 else:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1064 print(" it already exists, skipping")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1065 continue
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1066 if args.symlink == 'all':
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1067 os.symlink(path, dest)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1068 else:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1069 if path.is_dir():
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1070 shutil.copytree(path, dest)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1071 else:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1072 shutil.copyfile(path, dest)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1073
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1074 if args.version == 'dev':
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1075 print("exporting cache")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1076 dest = args.export / get_cache_dir()
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1077 if args.force_export and os.path.lexists(dest):
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1078 print(f" replacing existing {dest}")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1079 if dest.is_symlink():
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1080 dest.unlink()
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1081 else:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1082 shutil.rmtree(dest)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1083
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1084 if dest.exists():
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1085 print(" it already exists, skipping")
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1086 else:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1087 if args.symlink in ('all', 'cache'):
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1088 os.symlink(get_cache_dir().resolve(), dest)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1089 else:
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1090 shutil.copytree(get_cache_dir(), dest)
274af514a5cf flatpak: reworked packages + made a building script: Goffi <goffi@goffi.org> parents: diff changeset	1091
161 4fe61dcd856e flatpak: update build-manifest to retrieve dependencies without initial download phase: Goffi <goffi@goffi.org> parents: 157 diff changeset	1092 print("\nexport completed")

Mercurial > sat_docs

annotate flatpak/build_manifest.py @ 175:73053fcc74c2