Page tree
Skip to end of metadata
Go to start of metadata

Upload Architecture

Domains 

The main domains used during upload are:

AbstractImage: An image on Cytomine database. An image could be add to many projects,
Project: A workspace where you can view and use images,
ImageInstance: An instance of AbstractImage in a project (link between Project and AbstractImage),
Storage: Virtual directory where you can store a file,
UploadedFile: A file being uploaded. Later, it will give rise to an AbstractImage,
Property: key-value that you can add to a Cytomine domain (AbstractImage, ImageInstance, Project,...)

 

For more information regarding the package Image/Storage, see Package Image - Storage page.

Paradigm

1 uploaded file = 1 UploadedFile object except in these situations :

  • 1 Archive file. We decompress and reupload the content. The original file is tagged as the parent of the last UploadedFile
  • 1 file which need an heavy conversion. We reupload the result of conversion. The convertible file is tagged as the parent of the converted UploadedFile.

After these processes, an AbstractImage is created from the "last" UploadedFile. As conversion has been done, this AbstractImage reference a supported format. 

 

Upload Overview

The full upload architecture on Cytomine contains:
CORE: Cytomine main server (e.g. beta.cytomine.be, localhost-core)
IMS: Cytomine image management server (e.g. upload.cytomine.be, localhost-ims)
CLIENT: Cytomine client (Java, Python,...or the JS web client)
STORAGE/IMAGE SERVER: Server storing and delivering images
VIEWER JS: Cytomine js viewer

Complete upload procedure

1. Client send an HTTP message to IMS (POST IMS/upload) with image(s) and some metadata.
The main metadata are:

   
cytomineThe cytomine (Core) url where to add the image. An IMS may be used for multiple CoreMandatory
idStorageThe storage where to copy the imageMandatory
idProjectIf not null, link the image with this project (create a new ImageInstance from the AbstractImage)Optional, default null
syncSet the upload synchronizedOptional, default false
keysList of properties keys (join with “,”).
The key(i) will be link with values(i) and this key-value entry will be insert as a property for the image (AbstractImage).
values and keys must have the same number of items!
Optional, default null
valuesList of properties values (join with “,”). See keys definitionOptional, default nul

2. IMS checks parameters and authentify the user. The communication between Core and IMS is made trough the Java client. (StorageController)

3. IMS (UploadService) creates an UploadedFile instance on Cytomine-Core with the upload and copy it to its storage. The file path follows the convention storage_base_path/{current_timestamp}/original_filename.ext 

4. IMS (UploadService) tries to detect the format. A supported format is either

  • natively supported (Pyramidal TIFF, JPEG2000, OpenSlide supported formats)
  • supported after a light conversion (JPEG, PNG, BMP, DICOM, PlanarTIFF, HuronTIFF, ...). See step 6 for light conversion.
  • supported after a heavy conversion (all formats convertible to TIFF by BioFormat). See step 5 for heavy conversion.

If sync parameter is set to true, the conversion and deployment is done sequentially and not in background. In the case where sync=false, the next steps will be processed in background.

5. For the files which need a heavy conversion, they are converted using BioFormat. The result of conversion is reuploaed and the convertible file is tagged as the parent of the converted UploadedFile, as previously explained. Depending on the format, the process may take some time. 

6. For the files which need a light conversion, the format is convertible by VIPS and it creates a new pyramidal TIFF image, which is stored in the same directory as the original image, but this time with a random name (and the .tif extension).

7. IMS asks Core to create an AbstractImage image based on the UploadedFile and the novel corresponding pyramidal TIFF image. If a project was specified, Core will create and ImageInstance to make the link between this project and the freshly created AbstractImage.

8. IMS responds to the client. The reponse mainly contains: (200=OK),

AttributesDescription
statusHTTP response status. 200 = OK
uploadedFileThe uploaded file (JSON)
imagesA list of AbstractImage (generally with only 1 item).
This property is only available if sync parameter = true!
(if async, response is send before the AbstractImage creation)

 

Upload possible errors

Errors may happens at some step:

#Step summaryIf an error happens here...
Step 1Client send message to IMSThe client should gives you an error (network error,...)
Step 2-3IMS retrieve parameters and imageThe IMS should send you an error. It could be an auth error (bad keys) or parameters errors (keys.size!=values.size, bad project...)
If you are in async mode (by default, see sync parameter), IMS will response to the client after step 4. The reponse contains a property “uploadFile”. You can frequently get the uploaded file with a client to check the new status
Step 3->8IMS process and create image on CoreOnly if sync=true, you will get an error

 

Uploaded file status

Each UploadedFile has an associated status.

StatusNameDescription
0UPLOADEDDefault status when a new UploadedFile is added to Core (at step 3-4)
1CONVERTEDA file with heavy conversion has been converted (at step 5)
2DEPLOYEDThe file has been possibly converted by VIPS and a corresponding AbstractImage is created (from end of step 6)
3ERROR_FORMATThe format is not detected and the upload is aborted (at step 4)
4ERROR_CONVERTThere is an error during heavy conversion (at step 5)
5UNCROMPRESSEDFor archive format, but not used ?
6TO_DEPLOYThe format has been detected, waiting for deployment (between end of step 4 and step 6)

 

Related pages:

  • For more information regarding image/storage domain, see Package Image - Storage.

  • For more information regarding the IMS, see Part 7: IMS.
  • For more information regarding the image server (IIP server), see Part 8: Image server.
  • For more information regarding the image formats supported (natively, after light conversion or after heavy conversion), see Guide: Image format.

 


Upload files with a client.

Upload with Java

 

/**
* Upload and create an abstract image on the plateform (use async upload)
* @param file The image file path
* @param idProject If not null, add the image in this project
* @param idStorage The storage where the image will be copied
* @param cytomineHost The URL of the Core
* @return A response with the status, the uploadedFile and the AbstractImage list
* @throws Exception Error during upload
*/

public JSONArray uploadImage(String file,Long idProject,Long idStorage, String cytomineHost) throws Exception
/**
* Upload and create an abstract image on the plateform (use async or sync upload depending on synchrone parameter)
* @param file The image file path
* @param idProject If not null, add the image in this project
* @param idStorage The storage where the image will be copied
* @param cytomineHost The URL of the Core
* @param properties These key-value will be add to the AbstractImage as Property domain instance
* @param synchrone If true, the response will be send from server when the image will be converted, transfered, created,...(May take a long time)
* Otherwise the server response directly after getting the image and the parameters
* @return A response with the status, the uploadedFile and the AbstractImage list (only if synchrone!=true)
* @throws Exception Error during upload
*/
public JSONArray uploadImage(String file,Long idProject,Long idStorage, String cytomineHost, Map<String,String> properties, boolean synchrone) throws Exception

 

If you use async mode, you may want to retrieve the upload status

/**
* Get the uploaded file
* @param id Uploaded file id
*/
public UploadedFile getUploadedFile(Long id) throws Exception

 

Upload with Python

TODO.

See example (outdated?) here: https://github.com/cytomine/Cytomine-python-client/blob/master/client/examples/upload_image.py

 

  • No labels