README.md 2.69 KB
Newer Older
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
1
2
# Common CI/CD routines re-used in [CC Workflows](https://gitlab.astro.unige.ch/integral/cc-workflows)

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
3
4
|  | |
| ------ | ------ |
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
5
6
| **Prepared-For** | Developers of Cross-Calibration Test Cases: expert scientists, domain specialists |
| **Prepared-By** | Maintainer of [Cross-Calibration ToolKit](https://gitlab.astro.unige.ch/savchenk/ci-template-cc): [VS](https://gitlab.unige.ch/Volodymyr.Savchenko) |
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
7
8
| **See Also** | https://redmine.astro.unige.ch/projects/isgri-calibration/wiki/Common_ISGRI_Cross-Calibration_Workflows |

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
9

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
10
## Explanations
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
11

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
12
The workflows are mostly of INTEGRAL docker containers, which are [publicly available](https://hub.docker.com/orgs/integralsw). 
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
13

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
14
### Makefile:
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
15
16
17
18

Everybody likes to type "make". Also the make's target completing tracking is great and is a progenitor of similar techniques in other workflow mangment systems. But Makefile systax is another level of quoting hell, so all processes are actually described in [make-functions.sh](make-functions.sh)
In general, for every "make XXX" there is an "XXX" bash function in [make-functions.sh](make-functions.sh).

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
19
### make-functions.sh
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
20

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
21
#### *build* 
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
22
23
24
25
26

build the container image, if needed. 

### *run-one*

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
27
runs all notebooks in a loop, using [oda evaluation](https://github.com/cdcihub/oda-kb).
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
28
29
30
31
32
33
34
35
36
37
38
39

needs access to [ODA KB](https://github.com/volodymyrss/oda-kb), uses the default ODA_SPARQL_ROOT=http://fuseki.internal.odahub.io/dataanalysis

benefits from access to ODA DataLake (minio) - not criticial hence TBD.

Workflow input parameters can be set as:

```bash
$ NBARGS_PYDICT='{"nscw":2}' make run-one
```

note taht NBARGS_PYDICT should be a valid python dictionary.
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
40

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
41
Use **nbinspect** to list notebook parameters.
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
42

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
43
#### *run*
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
44
45
46

runs jupyter server, allowing to manually run and inspect the notebook. the notebook is mounted from the current directory hence can be later commited.

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
47
### Workflow Repository configuraion: oda.yaml
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
48
49
50

useful modifications for each workflow repository is contained in oda.yaml.

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
51
## image
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
52
53
54

integralsw/osa-python is the *base* docker image

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
55
56
57
## nb2workflow

[nb2workflow](https://github.com/volodymyrss/nb2workflow) adapts notebooks as workflows, either [CWL](https://www.commonwl.org/) jobs or [OpenAPI](https://swagger.io/specification/) services.
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
58
59
60

extract of some of the functions useful here:

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
61
62
63
- **nbrun** *[nbname]* runs notebook in a one-shot way. This can be done for test locally, and it is also done inside CWL container job (e.g. on REANA)

- **nbinspect** *[nbname]* shows notebook parameters
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
64

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
65
- **nb2service** runs a discoverable service that can execute the notebook on demand.
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
66
 
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
67
- **nb2worker** builds a container, either one-shot (for CWL) or service
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
68
69
70
71
72
73
74


## Running workflows on REANA

```bash
$ cc-cli run-reana rev1787.yaml
```