Page tree
Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 125 Next »

How to easily deploy Cytomine 1.0.

Cytomine server uses many libraries but this page describe how to have a production instance of Cytomine ready to use with only four steps.

Once the server is installed, one can access Cytomine through a modern web browser.

Please, before further reading, pay attention to the requirements. This installation documentation is for an Ubuntu 14.04 LTS.  

Please note we use Docker which is a kind of lightweight virtualization platform, so you do not need to create virtual machines to install Cytomine (it's better not do so for better performances).

If you experiment any issue, please contact us by e-mail or describe your problem precisely in our ticket system on Github.

 

Step 1) First of all, we retrieve the Cytomine-bootstrap installation procedure. Copy/paste these commands:

mkdir Cytomine_src/
cd Cytomine_src/
wget https://github.com/cytomine/Cytomine-bootstrap/archive/v1.0.zip
unzip v1.0.zip

 

Step 2) Then, we need to install the Docker engine. Copy/paste these commands (got from the official site) :
For other operating systems, please check at the Docker website (e.g. for CentOS see our installation notes). 

sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
sudo sh -c "echo deb https://apt.dockerproject.org/repo ubuntu-trusty main > /etc/apt/sources.list.d/docker.list"
sudo apt-get update
sudo apt-get install linux-image-extra-$(uname -r)
sudo apt-get update
sudo apt-get install docker-engine
sudo service docker start

 

Step 3) Now, it is required to edit the Cytomine file configuration.sh in the Bootstrap/ directory to prepare the installation of Cytomine Docker containers, using e.g.:

cd Cytomine-bootstrap-1.0/
emacs configuration.sh

In this file you might modify the variables "XXX_URL" by the desired Cytomine URLs, i.e. the URL dedicated to the Core of Cytomine must be set in the CORE_URL variable and so on. Please use URLs that are not already present in your /etc/hosts file to avoid conflicts.

This file also contains some arrays of values (e.g. : IMS_URLS to use multiple image servers). Be careful that their value are space free even between two values (example : IMS_URLS="[localhost-ims,localhost-ims2]" is correct and IMS_URLS="[localhost-ims, localhost-ims2]" is not). 

It is then mandatory to edit the variables "IMS_STORAGE_PATH", "IMS_BUFFER_PATH" by the corresponding existing paths on an available filesystem where image files will be stored. Please note you have to specify paths without a "/" at the end (e.g. "/data", not "/data/")

In the same way, backups will be stored in BACKUP_PATH, machine learning models in MODELS_PATH and some retrieval data in RETRIEVAL_PATH.

Note that you might also want to edit the default passwords (e.g. variables "RABBITMQ_PASS", "MEMCACHED_PASS",...) and check other options. An explanation of these variables is below.

Explanation of some variables :

Currently, in the configuration.sh file, we have variables used in the run command of the containers.

There are mandatory variables:

  • XXX_WAR_URL/XXX_JAR_URL/XXX_DOC_URL : URL of the WAR/JAR/DOC for the main parts of Cytomine (Core, IMS, retrieval, ...).

  • XXX_URL : URL to connect to the corresponding part of Cytomine (CORE, IMS, UPLOAD, RETRIEVAL, ...).

  • XXX_PATH : paths for buffer, permanent storage, machine learning models, etc. These paths must pre-exists.

  • XXX_PASS : default passwords (MEMCACHED).

There are optional variables:

  • IS_LOCAL : boolean. True if you deploy cytomine on a localhost.
  • GLUSTER_SERVER : (Doesn't work ! Let it blank). Let HAS_GLUSTER at false.

  • VOLUME : (Doesn't work ! Let it blank).

  • BACKUP_BOOL : boolean. True if you want automatic backups. In this case, the following variables are mandatory.
    • BACKUP_PATH : a directory with this name will be created in the DB container for "auto-backup".
    • SENDER_EMAIL, SENDER_EMAIL_PASS, SENDER_EMAIL_SMTP : email params of the sending account for report of the backup.
    • RECEIVER_EMAIL : email adress of the backup reports receiver.
  • BIOFORMAT_ENABLED : Enable the bioformat convertor to support more formats (VSI, OME-TIFF,...).

  • IRIS_ENABLED : Enable IRIS (see IRIS project)
  • IRIS_ID : ID. Usefull if we are admin of multiple IRIS version.
  • IRIS_ADMIN_EMAIL : The email adress where user will ask permissions for IRIS projects. 

If you deploy on a local host :

First of all, the variables "XXX_URL" will not be visible outside of your local server. So, due to the Docker architecture (isolation of the Docker's containers), we need to apply some changes to authorize network communication between our IMS and CORE containers (see below for further details). To do so, add these URL values (CORE_URL, IMS_URLS, UPLOAD_URL, RETRIEVAL_URL) to your /etc/hosts file with the following format : "127.0.0.1       XXX_URL".

With the default URL your etc/hosts will contain the following lines

127.0.0.1	localhost-core
127.0.0.1	localhost-ims
127.0.0.1	localhost-ims2
127.0.0.1	localhost-upload
127.0.0.1	localhost-retrieval

 

In the file configuration.sh, we will also need to set the variable IS_LOCAL at true.

NB : Please be careful that some problems might appear if you use localhost instead of CORE_URL for a connection to Cytomine. So, keep using CORE_URL.

If you want to make your Cytomine instance accessible from anywhere :

Contact your institutional system/network administrator before installing Cytomine so that they create DNS entries and make their HTTP port (80) accessible for CORE_URL, IMS_URLS, UPLOAD_URL, and RETRIEVAL_URL. In this case, in configuration.sh file, set IS_LOCAL = false before executing it.

 

Step 4) See page Known problems of v1.0 (and specially the entry "Error : postgis_data not found/not started").


Step 5) Then, you can deploy Cytomine by launching these three following commands in the Bootstrap directory (the creation of docker images can take up to 2 hours depending on network and server speed):

sudo sh ./clean_docker.sh #(the first time, you don't need to run this line)
sudo sh ./create_docker_images.sh
sudo sh ./start_deploy.sh

Please note that in some environments problems arise with network configuration (firewall) preventing docker to retrieve required packages from Internet. If docker fails to install the base container with error messages related to archive.ubuntu.com, you should first clean docker containers (using the first script above then the following command: sudo docker rmi $(sudo docker images -q) ) ; then you have to edit the DNS in the default docker configuration file and restart docker service as explained here: http://stackoverflow.com/questions/24151129/docker-network-calls-fail-during-image-build-on-corporate-network

Please also note that server components might take one or two minutes to boot the first time (which will generate a "502 Bad Gateway message" or the browser to wait for a reponse). 

At the end of the installation we recommend to install test data (which takes roughly 30 additional minutes) in order to follow our user guide.


 

How to test your installation

You can log into Cytomine through the http://CORE_URL in your web browser.

Please have look at our Cytomine User Guide for default usernames/passwords and see examples on toy data (You have to answer "yes" at the end of the installation procedure to inject these toy data).

To add new users, you can log using the admin account (password is asked during installation). You can then open an admin session (top right menu) to have access to User Configuration options. 

 

In addition we also describe here how to test basic features of Cytomine with a command-line Java client (there will be no major differences for Python).

First of all, launch Cytomine in your browser, go in your Account page (click on the user icon and on "Account") and save your public and private keys for the Java client.

Then, get the URL of the image server (IMS). We will also need the idStorage which can be found at the link http://CORE_URL/api/storage.json. 

Below we propose a template of a simple Groovy script with methods useful to add a project, add an ontology, upload an image and list annotations within an image.

1) Download and install the Cytomine Java Client on your local computer:

sudo apt-get install maven
sudo apt-get install -y software-properties-common
sudo add-apt-repository -y ppa:webupd8team/java
sudo apt-get -y update
sudo apt-get install -y oracle-java8-installer
sudo apt-get install groovy
 
cd Cytomine_src/
git clone https://github.com/cytomine/Cytomine-java-client.git
cd Cytomine-java-client/
mvn -U clean install
 
mkdir test
cp target/cytomine-java-client-1.0-SNAPSHOT-jar-with-dependencies.jar ../../test/
cd ../../test/
 

 

Then copy the content below into script.groovy file and run it with:

groovy -cp 'Cytomine-client-java.jar'  script.groovy
Groovy script
//run me with 
//groovy -cp 'Cytomine-client-java.jar'  script.groovy
import be.cytomine.client.*;
import be.cytomine.client.collections.*;
import be.cytomine.client.models.*;
String publickey = "";
String privatekey = "";
String cytomineCoreUrl = "";  // http://CORE_URL
String cytomineImsUrl = "";   // http://IMS_URL

println "********************************************************************"
println "Launch Test:"
Cytomine cytomine = new Cytomine(cytomineCoreUrl, publickey, privatekey, "./");

println "Try to add an ontology"
Ontology ont = cytomine.addOntology("testAddOntology")
println "Ontology created : "
println "	name : " + ont.get("name")
println "	id : " + ont.getId()
println "Try to add a project"
Project proj = cytomine.addProject("testAddProject", ont.id)
println "Project created : "
println "	name : " + proj.get("name")
println "	ontology : " + proj.get("ontology")
println "	id : " + proj.getId()
println "********************************************************************"
println "List your project:"
ProjectCollection projects = cytomine.getProjects();
for(int i=0;i<projects.size();i++) { 
	Project project = projects.get(i)
	println project.getId() + " " + project.get("name") + " (" + project.getLong("numberOfImages") + " images)" 
}
String imagePATH = ""; // fill here the absolute path of an image.
Long idStorage = ; // fill here the storage id from http://CORE_URL/api/storage.json

Cytomine cytomineUpload = new Cytomine(cytomineImsUrl+"/", publickey, privatekey, "./");
println cytomineUpload.uploadImage(imagePATH, proj.getId(), idStorage, cytomineCoreUrl, null, true);

println "********************************************************************"
println "List the images of your project:"
ImageInstanceCollection images = cytomine.getImageInstances(proj.getId());
for(int i=0;i<images.size();i++) { 
	ImageInstance image = images.get(i)
	println image.getId() 
}

println "********************************************************************"
println "List the annotations of your project:"
AnnotationCollection annotations = cytomine.getAnnotationsByProject(proj.getId());
for(int i=0;i<annotations.size();i++) {
	Annotation annotation = annotations.get(i)
	List termNameList = annotation.getList("term").collect{ termId ->
		return terms.list.find{it.id==termId}.get("name") 
	}
	println annotation.getId() + " " + annotation.get("location") + " " + annotation.get("image") + " " + annotation.get("name") + " " + annotation.get("cropURL") + (termNameList.isEmpty()? "" : " has term ${termNameList.join(",")}")
}

 


 

How to debug your installation.

We provide a script (in the Bootstrap directory) that outputs logs of the different containers and packs them in an reporting.tgz file for further analysis:

sudo sh ./reporting.sh

If you want to analyze more deeply an issue, depending on your problem (e.g. issue with image viewing), you can enter into the corresponding Docker container (e.g. ims) using the docker enter command and analyze log files.

First, install once the docker enter tool using:

docker run -v /usr/local/bin:/target jpetazzo/nsenter

 Then enter into the corresponding docker (e.g. image server) and analyze the log:

sudo docker-enter ims
tail -n 100 -f var/lib/tomcat7/logs/catalina.out

Here is a list of log file paths in the different docker containers (sudo docker logs container):

Cytomine ComponentDocker containerlog file
Cytomine-Corecore
var/lib/tomcat7/logs/catalina.out
Image serverims
var/lib/tomcat7/logs/catalina.out
IIPImage serveriipOff, iipJ2, iipCyto, iipVent
tmp/iip-openslide.out
Database serverdb
var/log/postgresql/postgresql-9.3-main.log
MongoDBmongodb
var/log/mongodb/mongo.log
Cytomine-DataMiningsoftware_router/software_router/algo/logs/

 

To debug from the Java client, create a log4j.properties file in the directory where you execute your script with the following content (INFO/WARN/ERROR/DEBUG provides different levels of messages):

#######################
# Root logger option
log4j.rootLogger=DEBUG, stdout
 
# Direct log messages to stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
#######################

 


How to update your installation.

Docker is also used to easily update your Cytomine installation.

For example, if 

  • We update the current WAR's, 
  • A newer version of a component is available (e.g. the image server to support newer image formats),
  • You customize some Docker images,

all you need to do is to restart the containers to have an up-to-date version of Cytomine with your own customization.

Of course, restarting all the containers will make you lost all your data. To avoid this problem, some containers are specifically designed for your data and must always be running (if you need to stop them, do a backup first) !

The restart_keep_data.sh script is made to re-deploy an up-to-date app without erasing your data.

So, as much as possible(*), only run the restart_keep_data.sh script to redeploy your app.

NB : If you add a custom container to your installation, you have to update this script.

* : The "data containers"  will have to be updated only if you made fundamental changes on the DBMS (e.g. update version, change postgres to mysql, ...)

 


How to uninstall Cytomine.

Please let us know why Cytomine does not meet your expectations.

In order to uninstall it, you have to delete all Docker container images using the following commands. 

!!! WARNING !!!: This will definitively delete all Cytomine data and all Docker containers.

!!! WARNING !!!: If you are using Docker for other projects, you have to delete only Cytomine containers (sudo docker images ; then for each Cytomine container: sudo docker rmi $id)

sudo sh clean_docker.sh
sudo docker rmi $(sudo docker images -q)

You should also delete remaining files in your local directories IMS_STORAGE_PATH, IMS_BUFFER_PATH, and BACKUP_PATH.

 

 

 

 

  • No labels