Mercurial > libervia-backend

annotate doc/jp/input.rst @ 3439:b166a59b1724

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc
author Goffi <goffi@goffi.org>
date Fri, 04 Dec 2020 12:37:38 +0100
parents 72583524cfd3
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3041
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
1 ================================================
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
2 input: automatise commands using external inputs
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
3 ================================================
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
4
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
5 ``input`` is a way to use external data (like file in a specific format) as input
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
6 arguments. It may be seen as a something similar to ``--output`` but for inputs.
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
7
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
8
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
9 csv
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
10 ===
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
11
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
12 CSV (for Comma-Separated Values) is a common format for tabular data. It is widely used in
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
13 spreadsheet software (at least at en export format). With ``csv`` command, you can use
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
14 columns a CSV file as arguments to jp commands.
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
15
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
16 To set the command, you'll write in sequence what to do with each column of your data.
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
17 For each column you can:
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
18
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
19 - specify a short option name using ``-s ARGUMENTS, --short ARGUMENTS`` (short options are
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
20 the ones with a single ``-``)
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
21 - specify a long option name using ``-l ARGUMENTS, --long ARGUMENTS`` (long options are
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
22 the ones with two ``-``)
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
23 - specify a positional argument using ``-p ARGUMENTS, --positional ARGUMENTS``
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
24 - indicate to use the column data with ``stdin`` using ``-i, --stdin``
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
25 - ignore the column if it's not used in the jp command, using ``-x, --ignore``
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
26
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
27 After each column specification, you may use a filter to manage the value. So far the
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
28 following filters are available:
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
29
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
30 ``-S, --split``
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
31 This will split the value (on any whitespace character, discarding empty values) and
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
32 repeat the option which each item. This is useful when you are using an option which can
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
33 be repeated (like ``-t TAG, --tag TAG`` with ``jp blog set``).
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
34
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
35 ``-E ARGUMENTS, --empty ARGUMENTS``
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
36 Indicate what to do if the column value is empty (by default en empty string is used).
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
37 You can use either ``skip`` to skip the whole row, or ``ignore`` so the option will not
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
38 be set at all (which is different from the default which will still set the option but
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
39 with an empty string).
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
40
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
41 CSV file is read from stdin, and by default unicode is expected. You may force an encoding
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
42 by using ``--encoding ENCODING``.
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
43
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
44 By default all the rows are read, but you may want to ignore first rows (if they are used
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
45 for columns title, or if you have already handled part of the list). To do that, use the
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
46 ``-r ROW, --row ROW`` option.
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
47
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
48 When you test your command, it is better to do a dry run to see what will happen. The
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
49 ``-D, --debug`` option is here for that: if you set it, the commands won't be actually
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
50 run, but the exact command which would be executed will be printed on screen. You should
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
51 always use this option first until you're sure that what you want will be executed.
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
52
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
53 You may add verbosity level to help debugging. With a verbosity level of 2 (i.e. ``-vv``)
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
54 the value read from CSV will be printed.
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
55
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
56 By default stdout and stderr of each launched command is ignored, but you can log them to
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
57 files using respectively ``--log LOG`` and ``--log-err LOG_ERR`` where ``LOG`` and
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
58 ``LOG_ERR`` are paths to a log file to create.
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
59
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
60 Once all the sequence and options are set, you write the jp command that you want to use,
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
61 with all the needed static option (i.e. options which must be used each time).
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
62
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
63
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
64 example
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
65 -------
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
66
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
67 Louise as a spreadsheet with a table like this:
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
68
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
69 ============================ ============ ============= ===============
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
70 title body internal data tags
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
71 ============================ ============ ============= ===============
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
72 Some title a body ABC jp demo
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
73 Something else another body XYZ jp demo
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
74 Third one third body VWZ special_tag jp
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
75 This one doesn't have a body 123 jp demo numbers
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
76 last one last body 456 jp demo numbers
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
77 ============================ ============ ============= ===============
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
78
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
79 She wants to use it as input data to create blog posts.
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
80
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
81 She first saves the file using CSV format, let's say to ``~/blog_input.csv``.
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
82
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
83 Then she checks ``jp blog set --help`` to get name of options to use. She'll need to use
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
84 ``--title`` for title, ``stdin`` for the body and ``-t`` for tags. Louise wants to
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
85 activate comments, so she also wants to use ``-C`` for all posts, and a tag to says it's
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
86 coming from the spreadsheet (using ``-t spreadsheet``) .
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
87
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
88 The first row of the table is used for columns headers, so she'll start at row 1 with ``-r
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
89 1``.
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
90
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
91 There is one row without body, Louise want to skip any row without body so she'll use the
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
92 ``-E skip`` filter, just after specifying the body row.
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
93
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
94 Reading column by column, the sequence is like this:
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
95
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
96 ``-l title``
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
97 a title which goes to the ``--title`` long option of ``jp blog``
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
98 ``-i -E skip``
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
99 a body which goes to the stdin of ``jp blog``. If the body is empty, the ``-E skip``
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
100 filter tells to skip the whole row.
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
101 ``-x``
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
102 the ``internal data`` column is not used, so it is ignored
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
103 ``-s t -S``
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
104 the last column are the tags, so the ``-t`` short option is used. There are several of
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
105 them separated by spaces, so the ``-S`` filter is used to split the values.
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
106
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
107 First she'll use the ``-D, --debug`` to check that the commands which will be executed are
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
108 the expected one::
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
109
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
110 $ jp input csv -D -r 1 -l title -i -E skip -x -s t -S blog set -C -t spreadsheet < ~/blog_input.csv
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
111
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
112 Everything seems fine, so she'll actually launch the command by running the same command
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
113 line but without the ``-D`` option::
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
114
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
115 $ jp input csv -r 1 -l title -i -E skip -x -s t -S blog set -C -t spreadsheet < ~/blog_input.csv
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
116
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
117 She could also have used ``--log`` and ``--log-err`` to check the logs of each command::
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
118
72583524cfd3 doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff changeset
119 $ jp input csv -r 1 -l title -i -E skip -x -s t -S --log /tmp/jp_blog_stdout.log --log-err /tmp/jp_blog_stderr.log blog set -C -t spreadsheet < ~/blog_input.csv