Page tree
Skip to end of metadata
Go to start of metadata

Manage processing servers

Each machine on which an algorithm can be executed is represented by a processing server.

Only Cytomine administrators are able to manage processing servers. You can have a list of servers with different abilities (CPU, GPU, simple server, HPC clusters, ...)

As Cytomine communicates with processing servers using SSH protocol, most of HPC servers can be easily added to Cytomine. There is nothing to install on the distant server. Only requirements are 

  • Singularity available on the distant platform. Singularity is a container management system (such as Docker) especially designed for HPC.
  • a compatible job scheduler on the distant platform. Currently supported job shedulers are
    • SLURM
    • No scheduler (the job is directly run on the shell)

From a credentials point of view, you have to

  • own an account on this processing server
  • copy the SSH public key of Cytomine in the $HOME/.ssh of the distant server. The SSH public key of Cytomine is the file id_rsa.pub in the directory SERVER_SSHKEYS_PATH of your configuration.sh 

Then add the new processing server to Cytomine (currently only available via the Java or Python client) to make this new machine visible for the execution of algorithms. You need to provide

  • the processing server name that will be displayed in Cytomine
  • the server host (IP address)
  • the username account on the distant machine
  • the port for SSH (in general: 22)
  • the type of server (informative) such as CPU or GPU
  • the working directory path, i.e. the absolute directory path for temporary data on the processing server (in general: the $HOME of username account on the processing server). Leave blank for $HOME
  • the persistent directory path, i.e. the absolute directory path for persistent data (built software images) on the processing server (for some servers a mass storage is provided to store large files outside of /home). Leave blank for $HOME
  • the processing method used. This parameter must be equal to a processing method name available in the software router. Currently "SlurmProcessingMethod" is supported.

In Python:

from cytomine import Cytomine
from cytomine.models import ProcessingServer
Cytomine.connect("host", "publicKey", "privateKey")
ProcessingServer(name="My super server", host="localhost", username="myusername", port=22, type="CPU", 
	persistent_directory="", working_directory="", processing_method_name="SlurmProcessingMethod").save()


By default, a simulated processing server is installed on the Cytomine-core server. It is thus possible to execute a software without installing any distant processing server. However, efficiency may be limited since Cytomine-core hardware is generally designed for a web server.


It is currently impossible to know from the Cytomine interface if a distant server is available nor the current server utilization. We plan however to retrieve such information and display them in Cytomine in the future, making it a large user-friendly front-end for a set of HPC clusters.

It is also not possible to restrict the usage of a processing server to a subset of Cytomine users.

Manage software user repositories

Cytomine is able to automatically retrieve source code and execution environment of a software from a VCS (version control system) and a Docker registry. To do so, you have to add some user repositories (userspace or organization) to your Cytomine instance, indicating that you trust them. Cytomine will periodically check these trusted repositories and update them if a new version is available.

If you have a Github orgnization where Cytomine software repositories are mixed with other repos, you can specify a prefix for each Cytomine software repo and only these ones will be checked. For example, Cytomine-ULiege Github organization prefixes all its software by "S_".

Only Cytomine administrators are able to manage software user repositories. 

Add a new user repository to trust it in your Cytomine instance with a client. You need to provide:

  • a VCS provider (currently only Github supported)
  • a VCS username (userspace or organization)
  • a Docker username
  • an optional prefix

In Python:

from cytomine import Cytomine
from cytomine.models import SoftwareUserRepository
Cytomine.connect("host", "publicKey", "privateKey")
SoftwareUserRepository(provider="Github", username="Cytomine-ULiege", docker_username="cytomineuliege", prefix="S_").save()

Manage available software of a Cytomine instance

If a new software is available on one of your trusted VCS repositories (a completely new one or a new release), it will be automatically added into Cytomine after a certain time. If you want to force the refresh of software user repositories, connect you as Cytomine admin (or super-admin) on a Web browser and then go to URL, where you use appropriate Cytomine-core URL (such as research.cytomine.be)

http://_YOUR_CYTOMINE_CORE_URL/api/software_user_repository/refresh_user_repositories.json

In the future, it will be directly available from the web interface for administration.

  • No labels