README.md 2.15 KB
Newer Older
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
1
2
# Common CI/CD routines re-used in [CC Workflows](https://gitlab.astro.unige.ch/integral/cc-workflows)

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
3
4
5
6
|  | |
| ------ | ------ |
| **See Also** | https://redmine.astro.unige.ch/projects/isgri-calibration/wiki/Common_ISGRI_Cross-Calibration_Workflows |

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
7
The workflows are mostly of INTEGRAL docker containers, which are [publicly available](https://hub.docker.com/orgs/integralsw). 
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
8

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
9
10
11
12
13
14
15
## Makefile:

Everybody likes to type "make". Also the make's target completing tracking is great and is a progenitor of similar techniques in other workflow mangment systems. But Makefile systax is another level of quoting hell, so all processes are actually described in [make-functions.sh](make-functions.sh)
In general, for every "make XXX" there is an "XXX" bash function in [make-functions.sh](make-functions.sh).

## make-functions.sh

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
### *build* 

build the container image, if needed. 

### *run-one*

runs notebook, using [oda evaluation](https://github.com/cdcihub/oda-kb).

needs access to [ODA KB](https://github.com/volodymyrss/oda-kb), uses the default ODA_SPARQL_ROOT=http://fuseki.internal.odahub.io/dataanalysis

benefits from access to ODA DataLake (minio) - not criticial hence TBD.

Workflow input parameters can be set as:

```bash
$ NBARGS_PYDICT='{"nscw":2}' make run-one
```

note taht NBARGS_PYDICT should be a valid python dictionary.
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
35

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
36
Use **nbinspect** to list notebook parameters.
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
37

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
38
## Workflow Repository configuraion: oda.yaml
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
39
40
41

useful modifications for each workflow repository is contained in oda.yaml.

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
42
## image
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
43
44
45

integralsw/osa-python is the *base* docker image

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
46
47
48
## nb2workflow

[nb2workflow](https://github.com/volodymyrss/nb2workflow) adapts notebooks as workflows, either [CWL](https://www.commonwl.org/) jobs or [OpenAPI](https://swagger.io/specification/) services.
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
49
50
51

extract of some of the functions useful here:

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
52
53
54
- **nbrun** *[nbname]* runs notebook in a one-shot way. This can be done for test locally, and it is also done inside CWL container job (e.g. on REANA)

- **nbinspect** *[nbname]* shows notebook parameters
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
55

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
56
- **nb2service** runs a discoverable service that can execute the notebook on demand.
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
57
 
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
58
- **nb2worker** builds a container, either one-shot (for CWL) or service