README.md 1.57 KB
Newer Older
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
1
2
# Common CI/CD routines re-used in [CC Workflows](https://gitlab.astro.unige.ch/integral/cc-workflows)

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
3
The workflows are mostly of INTEGRAL docker containers, which are [publicly available](https://hub.docker.com/orgs/integralsw). 
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
4

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
5
6
7
8
9
10
11
12
13
14
15
## Makefile:

Everybody likes to type "make". Also the make's target completing tracking is great and is a progenitor of similar techniques in other workflow mangment systems. But Makefile systax is another level of quoting hell, so all processes are actually described in [make-functions.sh](make-functions.sh)
In general, for every "make XXX" there is an "XXX" bash function in [make-functions.sh](make-functions.sh).

## make-functions.sh

- *build* build the container, if needed

- *run-one* runs notebook, using [oda evaluation](https://github.com/cdcihub/oda-kb).

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
16
17
18
19
## Repository configuraion: oda.yaml

useful modifications for each workflow repository is contained in oda.yaml.

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
20
## image
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
21
22
23

integralsw/osa-python is the *base* docker image

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
24
25
26
## nb2workflow

[nb2workflow](https://github.com/volodymyrss/nb2workflow) adapts notebooks as workflows, either [CWL](https://www.commonwl.org/) jobs or [OpenAPI](https://swagger.io/specification/) services.
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
27
28
29

extract of some of the functions useful here:

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
30
31
32
- **nbrun** *[nbname]* runs notebook in a one-shot way. This can be done for test locally, and it is also done inside CWL container job (e.g. on REANA)

- **nbinspect** *[nbname]* shows notebook parameters
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
33

Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
34
- **nb2service** runs a discoverable service that can execute the notebook on demand.
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
35
 
Volodymyr SAVCHENKO's avatar
Volodymyr SAVCHENKO committed
36
- **nb2worker** builds a container, either one-shot (for CWL) or service