Cytomine Task

WARNING

App Engine and its Task system are still in BETA. Therefore, all information presented in this documentation is subject to potentially breaking changes.

A Task is a time-limited process that takes static inputs and produces static outputs represented by files and directories. A Task is defined by:

1. a docker image encapsulating the environment and procudure of the Task
2. a descriptor file providing both general and technical information about the Task itself

In order to upload a Task on Cytomine, it must be bundled as a zip archive containing the docker image and the descriptor file. The Docker image must be saved into a tar archive before being included in the bundle.

How to create a Task

1. Write your Task descriptor based on the Task descriptor reference
2. Implement your Task in such a way that the script or program running the Task can be containerised
3. Create a Dockerfile to build the Task container image
4. Build and save the Docker image as a tar archive, the name of the archive should be formatted following {namespace}-{version}.tar and matching configuration.image.file in descriptor
5. Bundle the saved Docker image and the Task descriptor into a zip archive

Run

To run a task in the app-engine it creates a context for the execution to manage inputs provisioning, outputs generation, the Run has different states tracking the flow of execution.

STATE	DESCRIPTION
CREATED	Created but not all inputs provisioned
PROVISIONED	Ready to be executed, all inputs have been provisioned
QUEUING	Submitting the task to the execution environment
QUEUED	Evaluated successfully by execution environment
RUNNING	Task running in the execution environment
FAILED	An error occurred and stopped the process of executing the task (terminal state)
PENDING	Pending execution on the execution environment
FINISHED	Task successfully executed and outputs available (terminal state)

Storage

The app-engine controls provisioning of inputs to tasks and outputs back to consumers like core in Cytomine, every Run has its separate storage that contains the provisioned inputs and the generated outputs, and tasks metadata files like descriptors or logos. Data is represented in storage following the definitions of parameter data types as defined in Task I/O

The task run directories can be accessed by tasks running in local mode mode by mounting them as volumes.

data/
...
├── app-engine/
│   ...
│   ├── task-acf56ae9-9c18-493d-b8cd-28be5abd9cf1-def/  #<-------- task definition
│       ├── descriptor.yml
│       └── logo.png
│   ├── task-run-inputs-3dd5f08f-e9b7-4279-9d38-9ac8a38610d1/ #<-------- task run inputs provisioned by cytomine
│       └── input
│   ├── task-run-outputs-3dd5f08f-e9b7-4279-9d38-9ac8a38610d1/ #<-------- task run outputs generated by the task after execution
│       └── output
│   ...
...