Mercurial > libervia-backend
annotate doc/jp/input.rst @ 3446:d2298ed6de7f
docker (pubsub): install latest version of `postgresql-client` with PostgreSQL own Debian repos.
author | Goffi <goffi@goffi.org> |
---|---|
date | Thu, 10 Dec 2020 10:28:05 +0100 |
parents | 72583524cfd3 |
children |
rev | line source |
---|---|
3041
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
1 ================================================ |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
2 input: automatise commands using external inputs |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
3 ================================================ |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
4 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
5 ``input`` is a way to use external data (like file in a specific format) as input |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
6 arguments. It may be seen as a something similar to ``--output`` but for inputs. |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
7 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
8 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
9 csv |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
10 === |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
11 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
12 CSV (for Comma-Separated Values) is a common format for tabular data. It is widely used in |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
13 spreadsheet software (at least at en export format). With ``csv`` command, you can use |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
14 columns a CSV file as arguments to jp commands. |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
15 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
16 To set the command, you'll write in sequence what to do with each column of your data. |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
17 For each column you can: |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
18 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
19 - specify a short option name using ``-s ARGUMENTS, --short ARGUMENTS`` (short options are |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
20 the ones with a single ``-``) |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
21 - specify a long option name using ``-l ARGUMENTS, --long ARGUMENTS`` (long options are |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
22 the ones with two ``-``) |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
23 - specify a positional argument using ``-p ARGUMENTS, --positional ARGUMENTS`` |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
24 - indicate to use the column data with ``stdin`` using ``-i, --stdin`` |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
25 - ignore the column if it's not used in the jp command, using ``-x, --ignore`` |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
26 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
27 After each column specification, you may use a filter to manage the value. So far the |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
28 following filters are available: |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
29 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
30 ``-S, --split`` |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
31 This will split the value (on any whitespace character, discarding empty values) and |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
32 repeat the option which each item. This is useful when you are using an option which can |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
33 be repeated (like ``-t TAG, --tag TAG`` with ``jp blog set``). |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
34 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
35 ``-E ARGUMENTS, --empty ARGUMENTS`` |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
36 Indicate what to do if the column value is empty (by default en empty string is used). |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
37 You can use either ``skip`` to skip the whole row, or ``ignore`` so the option will not |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
38 be set at all (which is different from the default which will still set the option but |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
39 with an empty string). |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
40 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
41 CSV file is read from stdin, and by default unicode is expected. You may force an encoding |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
42 by using ``--encoding ENCODING``. |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
43 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
44 By default all the rows are read, but you may want to ignore first rows (if they are used |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
45 for columns title, or if you have already handled part of the list). To do that, use the |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
46 ``-r ROW, --row ROW`` option. |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
47 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
48 When you test your command, it is better to do a dry run to see what will happen. The |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
49 ``-D, --debug`` option is here for that: if you set it, the commands won't be actually |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
50 run, but the exact command which would be executed will be printed on screen. You should |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
51 always use this option first until you're sure that what you want will be executed. |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
52 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
53 You may add verbosity level to help debugging. With a verbosity level of 2 (i.e. ``-vv``) |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
54 the value read from CSV will be printed. |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
55 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
56 By default stdout and stderr of each launched command is ignored, but you can log them to |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
57 files using respectively ``--log LOG`` and ``--log-err LOG_ERR`` where ``LOG`` and |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
58 ``LOG_ERR`` are paths to a log file to create. |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
59 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
60 Once all the sequence and options are set, you write the jp command that you want to use, |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
61 with all the needed static option (i.e. options which must be used each time). |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
62 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
63 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
64 example |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
65 ------- |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
66 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
67 Louise as a spreadsheet with a table like this: |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
68 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
69 ============================ ============ ============= =============== |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
70 title body internal data tags |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
71 ============================ ============ ============= =============== |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
72 Some title a body ABC jp demo |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
73 Something else another body XYZ jp demo |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
74 Third one third body VWZ special_tag jp |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
75 This one doesn't have a body 123 jp demo numbers |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
76 last one last body 456 jp demo numbers |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
77 ============================ ============ ============= =============== |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
78 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
79 She wants to use it as input data to create blog posts. |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
80 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
81 She first saves the file using CSV format, let's say to ``~/blog_input.csv``. |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
82 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
83 Then she checks ``jp blog set --help`` to get name of options to use. She'll need to use |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
84 ``--title`` for title, ``stdin`` for the body and ``-t`` for tags. Louise wants to |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
85 activate comments, so she also wants to use ``-C`` for all posts, and a tag to says it's |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
86 coming from the spreadsheet (using ``-t spreadsheet``) . |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
87 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
88 The first row of the table is used for columns headers, so she'll start at row 1 with ``-r |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
89 1``. |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
90 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
91 There is one row without body, Louise want to skip any row without body so she'll use the |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
92 ``-E skip`` filter, just after specifying the body row. |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
93 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
94 Reading column by column, the sequence is like this: |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
95 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
96 ``-l title`` |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
97 a title which goes to the ``--title`` long option of ``jp blog`` |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
98 ``-i -E skip`` |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
99 a body which goes to the stdin of ``jp blog``. If the body is empty, the ``-E skip`` |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
100 filter tells to skip the whole row. |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
101 ``-x`` |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
102 the ``internal data`` column is not used, so it is ignored |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
103 ``-s t -S`` |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
104 the last column are the tags, so the ``-t`` short option is used. There are several of |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
105 them separated by spaces, so the ``-S`` filter is used to split the values. |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
106 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
107 First she'll use the ``-D, --debug`` to check that the commands which will be executed are |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
108 the expected one:: |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
109 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
110 $ jp input csv -D -r 1 -l title -i -E skip -x -s t -S blog set -C -t spreadsheet < ~/blog_input.csv |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
111 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
112 Everything seems fine, so she'll actually launch the command by running the same command |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
113 line but without the ``-D`` option:: |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
114 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
115 $ jp input csv -r 1 -l title -i -E skip -x -s t -S blog set -C -t spreadsheet < ~/blog_input.csv |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
116 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
117 She could also have used ``--log`` and ``--log-err`` to check the logs of each command:: |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
118 |
72583524cfd3
doc (jp): jp commands are now fully documented:
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
119 $ jp input csv -r 1 -l title -i -E skip -x -s t -S --log /tmp/jp_blog_stdout.log --log-err /tmp/jp_blog_stderr.log blog set -C -t spreadsheet < ~/blog_input.csv |