README.md 2.33 KB
Newer Older
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
1
2
# Common CI/CD routines re-used in [CC Workflows](https://gitlab.astro.unige.ch/integral/cc-workflows)

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
3
4
5
6
|  | |
| ------ | ------ |
| **See Also** | https://redmine.astro.unige.ch/projects/isgri-calibration/wiki/Common_ISGRI_Cross-Calibration_Workflows |

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
7
The workflows are mostly of INTEGRAL docker containers, which are [publicly available](https://hub.docker.com/orgs/integralsw). 
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
8

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
9
10
11
12
13
14
15
## Makefile:

Everybody likes to type "make". Also the make's target completing tracking is great and is a progenitor of similar techniques in other workflow mangment systems. But Makefile systax is another level of quoting hell, so all processes are actually described in [make-functions.sh](make-functions.sh)
In general, for every "make XXX" there is an "XXX" bash function in [make-functions.sh](make-functions.sh).

## make-functions.sh

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
16
17
18
19
20
21
### *build* 

build the container image, if needed. 

### *run-one*

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
22
runs all notebooks in a loop, using [oda evaluation](https://github.com/cdcihub/oda-kb).
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
23
24
25
26
27
28
29
30
31
32
33
34

needs access to [ODA KB](https://github.com/volodymyrss/oda-kb), uses the default ODA_SPARQL_ROOT=http://fuseki.internal.odahub.io/dataanalysis

benefits from access to ODA DataLake (minio) - not criticial hence TBD.

Workflow input parameters can be set as:

```bash
$ NBARGS_PYDICT='{"nscw":2}' make run-one
```

note taht NBARGS_PYDICT should be a valid python dictionary.
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
35

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
36
Use **nbinspect** to list notebook parameters.
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
37

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
38
39
40
41
### *run*

runs jupyter server, allowing to manually run and inspect the notebook. the notebook is mounted from the current directory hence can be later commited.

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
42
## Workflow Repository configuraion: oda.yaml
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
43
44
45

useful modifications for each workflow repository is contained in oda.yaml.

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
46
## image
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
47
48
49

integralsw/osa-python is the *base* docker image

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
50
51
52
## nb2workflow

[nb2workflow](https://github.com/volodymyrss/nb2workflow) adapts notebooks as workflows, either [CWL](https://www.commonwl.org/) jobs or [OpenAPI](https://swagger.io/specification/) services.
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
53
54
55

extract of some of the functions useful here:

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
56
57
58
- **nbrun** *[nbname]* runs notebook in a one-shot way. This can be done for test locally, and it is also done inside CWL container job (e.g. on REANA)

- **nbinspect** *[nbname]* shows notebook parameters
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
59

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
60
- **nb2service** runs a discoverable service that can execute the notebook on demand.
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
61
 
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
62
- **nb2worker** builds a container, either one-shot (for CWL) or service