# Release a Cytomine App

To release a Cytomine app, you need to know how to write a Cytomine app.

Cytomine apps are complete programs manipulating Cytomine data, where every execution is reported to Cytomine, with the parameters applied to the program, and the results of each program execution can then be compared from the graphical interface.

By releasing a Cytomine app,

the app is made available to the Cytomine community
the app can be launched directly from the graphical interface in a user-friendly way, including by non-technician users
the app is run on Cytomine server (or on dedicated hardware, depending on your configuration)
the app is run in an isolated environment for result reproducibility
the app log is retrieved for every execution for result traceability

To release a Cytomine app, it is mandatory to define the execution environment and the app dependencies.

# Adapt the JSON descriptor

There are some new keys that need to be declared in the JSON descriptor to release the Cytomine app.

# Command line

When the Cytomine app is released, it is the Cytomine server that has to create the execution command line, to launch the execution. It is done with the command-line key, that has to be templated with placeholders to be generic.

From this real execution command line

python app.py --cytomine_host https://mycytomine.com --cytomine_public_key AAA --cytomine_private_key ZZZ --cytomine_id_project 42 --cytomine_id_software 123 --alpha 0.5

we can replace values with placeholders:

python app.py --cytomine_host [CYTOMINE_HOST] --cytomine_public_key [CYTOMINE_PUBLIC_KEY] --cytomine_private_key [CYTOMINE_PRIVATE_KEY] --cytomine_id_project [CYTOMINE_ID_PROJECT] --cytomine_id_software [CYTOMINE_ID_SOFTWARE] --alpha [ALPHA]

Next step is to associate command line flag and placeholders to their parameter in the JSON descriptor:

the command line flag --cytomine_host is associated to parameter cytomine_host by setting its command-line-flag key to --cytomine_host
the placeholder [CYTOMINE_HOST] is replaced by the value of the parameter cytomine_host by setting its value-key to [CYTOMINE_HOST]

Put together, we have:

{
    "name": "MyFirstCytomineApp",
    "description": "This Cytomine App is a first test.",
    "schema-version": "cytomine-0.1",
    "command-line": "python app.py --cytomine_host [CYTOMINE_HOST] --cytomine_public_key [CYTOMINE_PUBLIC_KEY] --cytomine_private_key [CYTOMINE_PRIVATE_KEY] --cytomine_id_project [CYTOMINE_ID_PROJECT] --cytomine_id_software [CYTOMINE_ID_SOFTWARE] --alpha [ALPHA]",
    "inputs": [
        {
            "id": "cytomine_host",
            "name": "Cytomine host",
            "set-by-server": true,
            "optional": false,
            "type": "String",
            "command-line-flag": "--cytomine_host",
            "value-key": "[CYTOMINE_HOST]"
        },
        // ... Other parameters
    ]
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

The command line flag is handy to declare named parameters. When set as empty, the parameter is thus a positional parameter in the command line, without a flag.

Very often the command line flag is the app parameter programmatic name (id) preceded by --. The value key placeholder is less constrained but, very often it is the app parameter programmatic name in uppercase encapsulated in square brackets.

To avoid too much repetition in the descriptor, id value can be inserted automatically in any attribute of the same app parameter (input) using placeholders:

Placeholder	Replaced by
`@id`	id value
`@ID`	id value in uppercase

The JSON descriptor can be rewritten as:

{
    "name": "MyFirstCytomineApp",
    "description": "This Cytomine App is a first test.",
    "schema-version": "cytomine-0.1",
    "command-line": "python app.py --cytomine_host [CYTOMINE_HOST] --cytomine_public_key [CYTOMINE_PUBLIC_KEY] --cytomine_private_key [CYTOMINE_PRIVATE_KEY] --cytomine_id_project [CYTOMINE_ID_PROJECT] --cytomine_id_software [CYTOMINE_ID_SOFTWARE] --alpha [ALPHA]",
    "inputs": [
        {
            "id": "alpha",
            "name": "Alpha",
            "description": "This is the alpha parameter",
            "type": "Number",
            "optional": false,
            "default-value": 0.5,
            "value-key": "[@ID]",
            "command-line-flag": "--@id"
        },
        {
            "id": "cytomine_host",
            "name": "Cytomine host",
            "set-by-server": true,
            "optional": false,
            "type": "String",
            "value-key": "[@ID]",
            "command-line-flag": "--@id"
        },
        {
            "id": "cytomine_public_key",
            "name": "Cytomine public key",
            "set-by-server": true,
            "optional": false,
            "type": "String",
            "value-key": "[@ID]",
            "command-line-flag": "--@id"
        },
        {
            "id": "cytomine_private_key",
            "name": "Cytomine private key",
            "set-by-server": true,
            "optional": false,
            "type": "String",
            "value-key": "[@ID]",
            "command-line-flag": "--@id"
        },
        {
            "id": "cytomine_id_project",
            "name": "Cytomine project ID",
            "set-by-server": true,
            "optional": false,
            "type": "Number",
            "value-key": "[@ID]",
            "command-line-flag": "--@id"
        },
        {
            "id": "cytomine_id_software",
            "name": "Cytomine software ID",
            "set-by-server": true,
            "optional": false,
            "type": "Number",
            "value-key": "[@ID]",
            "command-line-flag": "--@id"
        }
    ]
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63

# Container image

The last modification to the JSON descriptor is to indicate that the Cytomine app runs in a container image, that is an isolated environment with its dependencies. The container-image key takes a JSON object as value with:

type: the type of container image for execution
image: the name of the container image for the Cytomine app

The type has to be singularity as Cytomine only support Singularity as execution engine.

If the Cytomine app is released with a trusted source, the container image name has to follow some conventions.

Limitation

When the Cytomine app is released on a trusted source, image must be equals to Cytomine app name in lowercase, prefixed by the trusted source prefix in lowercase and preceded by container image index username.

For a trusted source with prefix S_ and container image index cytomine and a Cytomine app named MyAwesomeApp, the container image image is cytomine/s_myawesomeapp

# Release with trusted source

A trusted source is a trusted link between a cloud source-code space, and a cloud environment space.

Currently, only GitHub is supported as source-code provider and DockerHub as environment provider.

These trusted sources are frequently automatically fetched by the Cytomine platform, and every new Cytomine app releases are added to the Cytomine server. To filter Cytomine app repositories from others in a GitHub space, a prefix can be defined in the trusted source to only consider repositories starting by the prefix as a Cytomine app.

The trusted source has to be added at Cytomine server level by your Cytomine administrator. To add your Cytomine app to a trusted source, you need to :

have an account on GitHub
have an account on DockerHub
link your DockerHub account with your GitHub account (opens new window)
be a member of the GitHub space (user or organization) of the trusted source
be a member of the DockerHub space (user or organization) of the trusted source

Contact your Cytomine administrator if needed.

As a figure is a thousand words, here is the big picture of a Cytomine app release.

# Initialize repositories

In this guide, we release a new Cytomine app in the following trusted source as an example:

Source code repository provider: GitHub (opens new window)
Source code repository space: cytomine (opens new window)
Environment repository provider: DockerHub (opens new window)
Environment repository space: cytomine (opens new window)
Prefix: S_

# Source code repository

Currently, only GitHub is supported as source code repository provider.

Create a new repository in the GitHub space referenced by the trusted source. On GitHub, space is either your user space or a GitHub organization.

Repository name must:

have less than 40 characters
start by the trusted source prefix (here S_)
correspond to the Cytomine app name written in the JSON descriptor (preceded by the prefix)

In the example, we create a repository named S_SampleDetector as the trusted source prefix is S_ and SampleDetector the Cytomine app name in the JSON descriptor.

The repository is now created in your GitHub space. Follow GitHub instructions to import your existing source code (including the Dockerfile) to GitHub.

# Environment repository

This feature requires a Docker subscription!

The following tutorial requires a Docker paid subscription.

A free workaround is proposed in the Workaround section and is based on GitHub actions (opens new window).

We will now create a link between GitHub and DockerHub. The goal is to automatically build the Cytomine app Docker container from the source code on GitHub.

Go on DockerHub (opens new window) and log in
Click on Account Settings then on Linked Accounts
Click Connect at the right of the GitHub line. You will be redirected to GitHub authentication page to enable the link
You will be able to link your DockerHub repositories to GitHub ones. Return to Repositories then click on Create Repository

We will create the s_sampledetector dockerhub repository

Choose the right Docker namespace corresponding to the trusted source
Choose the Docker image name.

Lowercase

Docker requires image names to be in lowercase. DockerHub automatically converts your entry to lowercase.
Image name

It must:
- be the same as Cytomine App repository name (Cytomine app name in JSON descriptor, preceded by optional prefix) but in lowercase
- be the same as declared in JSON descriptor image-name.
If these conventions are not strictly followed, some issues may occur.

3. Click on the GitHub logo and configure when to trigger a Docker image build. Set Type to Tag, set Source to /^v([0-9.]+)$/ and set Docker Tag to {sourceref}
4. Click on Create

After the creation, you will still be able to add or modify an automatic build rules by clicking to the Builds panel then Link to GitHub

WARNING

The default build rule displayed by DockerHub use the Branch type and the master branch source by default ! Don't forget to modify it to set Type to Tag, Source to /^v([0-9.]+)$/ and Docker Tag to {sourceref} as the example below.
Be sure to only have a single build rule in your Build Setting after saving.

The Docker build settings are now configured. All this setup has to be done only once per Cytomine app. Each time you will release a new version of your Cytomine app, a new Docker image will be built automatically.

# Workaround

This section describes a workaround to have a free automated build with GitHub actions.

Copy the following snippet to your repository in a file named build-and-push-docker-image.yml
- This file must be located in the path .github/workflows inside your repository.
- Replace your_docker_image_name by the Docker image name.
- Push the file to your GitHub repository.
Image name

It must:
- be the same as Cytomine App repository name (Cytomine app name in JSON descriptor, preceded by optional prefix) but in lowercase
- be the same as declared in JSON descriptor image-name.
If these conventions are not strictly followed, some issues may occur.

name: Docker Image CI
on:
  release:
    types: [published]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout 
        uses: actions/checkout@v3
      - name: Login to Docker Hub
        uses: docker/login-action@v1
        with:
          username: ${{ secrets.DOCKER_HUB_USERNAME }}
          password: ${{ secrets.DOCKER_HUB_ACCESS_TOKEN }}
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      - name: Build and push
        uses: docker/build-push-action@v2
        with:
          context: .
          file: ./Dockerfile
          push: true
          tags: ${{ secrets.DOCKER_HUB_USERNAME }}/your_docker_image_name:${{ github.ref_name }}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

Generate a Docker Hub access Token
- Log in to your Docker Hub (opens new window) account
- In the top right corner, click on your account and go to Account Settings then Security
- Click on the button New Access Token then complete the Access Token Description and click on Generate
- Copy the generated token
Go to your GitHub repository
- Click on Settings
- In the section Security, click on Secrets, then Actions
- Create the secret for the Docker Hub username:
  - Click on New repository secret
  - Complete the name with DOCKER_HUB_USERNAME and the value with your Docker Hub username
  - Click on Add secret
- Create the secret for the Docker Hub access token:
  - Click on New repository secret
  - Complete the name with DOCKER_HUB_ACCESS_TOKEN and the value with the generated token in step 2.
  - Click on Add secret

The automated build settings are now configured. All this setup has to be done only once per Cytomine app. Each time you will release a new version of your Cytomine app, a new Docker image will be built automatically and the image will be pushed to your DockerHub automatically.

# Make a new release

When your Cytomine app is ready to be released, commit your changes to GitHub and create a new GitHub release:

As GitHub suggests, try to follow semantic versioning.

GitHub has created a new release for your Cytomine app. In fact, it added a new git tag to the last commit.

If the Auto-build is correctly configured on DockerHub, it triggers a new Docker image build. After some time (depends on DockerHub), the Docker image is made available on DockerHub.

All Cytomine servers having the right trusted source configured will automatically add this new Cytomine app release and make it runnable directly from Cytomine graphical interface.

If it is the first release of the Cytomine app, the app is made available in Algorithms page on the graphical interface.

If it is a new release of an already existing Cytomine app, this new version is made available in Algorithms page on the graphical interface, enabled in all Cytomine projects having a previous version of the app, and previous versions of the app are set as Deprecated.

# Release with direct upload

If, for some reason, you do not want to use code and environment repositories, you can add your software directly to the platform via a direct upload to the Core Cytomine component.
Please note that, in this case, the update and version management of your software will not be automatically made via Cytomine, but manually by explicitely mention your App version during your upload (only in the latest master branch of the Cytomine python client for the moment).

Currently, upload a software is only possible using the Cytomine Python client and the Cytomine Java client. A script has been added to the collection of examples availables for the Cytomine Python client : upload_software.py (opens new window)

Using this script you will be able to upload the .zip archive with all your files and descriptors to your favorite Cytomine instance. Exemple of command to use for that :

    python upload_software.py  --cytomine_host "YOUR-INSTANCE-URL" --cytomine_public_key "YOUR-PUBLIC-KEY" --cytomine_private_key "YOUR-PRIVATE-KEY" --software_name "Python Software Example" --software_version "v1.0.0" --filepath "S_python_software_example.zip"

All the parameters to consider are explained in the Python client page.

Regarding what is inside the .zip archive, there two solutions, depending on where does your App Docker container is built :

# If you want Cytomine to build your Docker container (default)

If this solution is chosen, Cytomine will build itself the Docker container of your App using the Dockerfile. This Docker container will be build inside the software_router.

This is the default solution as in the software_router configuration (Cytomine_bootstrap/configs/software_router/config.groovy) the variable cytomine.software.allowDockerfileCompilation is by default set to TRUE.

To upload your App using this solution create a zip archive with your script, the descriptor.json, the Dockerfile and all the files mandatory for the Dockerfile compilation (including your script file), and upload this zip file to your Cytomine the software_upload.py script using the command described hereupper.

IMPORTANT NOTE : building a Docker container inside an other Docker container (software_router) will only be efficient if your App Docker container is relatively small (less than 1.5 GB). If you App container is more than approximatively 1.5 GB you should prefer to use the next solution.

# If you want to upload a pre-build Docker container

If you App Docker container is more than approximatively 1.5 GB it is definitively better to build it on your own computer before you upload you App in Cytomine. To do that, you have to change the configuration of the software_router and export your Docker container as a .tar file.

# Configure the software_router

In the software_router configuration (Cytomine_bootstrap/configs/software_router/config.groovy) set the variable cytomine.software.allowDockerfileCompilation to FALSE, and restart your Cytomine.

# Export your App Docker container as image.tar

From inside the folder where your Dockerfile is, execute this command (my_image being the name of the container as specified in your descriptor.json, and version the actual version of your App) :

    sudo docker build -t my_image:version .

When your Docker image is built, export it as a .tar file using :

    sudo docker save my_image:version > image.tar

To upload your App using this solution create a zip archive with your script, the descriptor.json, the image.tar archive and all the files mandatory for your App to run and upload this zip file to your Cytomine the software_upload.py script using the command described hereupper.

← Write a Cytomine App JSON descriptor reference →