Demorun 1 Running prediction with pre trained model

Segmentation with pretrained model (for mitochondria in a SBEM dataset)

These steps skip the training and go straight to prediction by using a 30,000 iteration pretrained mitochrondria model that has been preloaded.

Step 1) Connect via ssh to instance created by these instructions

ssh ubuntu@PublicDNS_VALUEFROM_WEBPAGE_ABOVE

Quick and Dirty run

Here are the commands to run once logged into the instance. Below in Steps 2+ are more detailed instructions on what is happening. The commands below will take 15 minutes to run on p2.xlarge and even less on p3.2xlarge instance.

cd ~
PreprocessImageData.m ~/cdeep3m/mito_testsample/testset/ ~/mito_testaugimages
runprediction.sh ~/sbem/mitochrondria/xy5.9nm40nmz/30000iterations_train_out ~/mito_testaugimages/ ~/predictout30k
EnsemblePredictions.m ~/predictout30k/1fm ~/predictout30k/3fm ~/predictout30k/5fm ~/predictout30k/ensembled
# results are in ~/predictout30k/ensembled
ls ~/predictout30k/ensembled

Step 2) Run PreprocessImageData.m

This step converts the png image files in ~/cdeep3m/mito_testsample/testset/ to hdf5 format and for the small test dataset takes 10-20 seconds to run.

PreprocessImageData.m ~/cdeep3m/mito_testsample/testset/ ~/mito_testaugimages

Output:

octave: X11 DISPLAY environment variable not set
octave: disabling GUI features
Starting Image Augmentation
Check image size of:
/home/ubuntu/cdeep3m/mito_testsample/testset/
Reading file: /home/ubuntu/cdeep3m/mito_testsample/testset/images.040.png
warning: your version of GraphicsMagick limits images to 16 bits per pixel
Reading file: /home/ubuntu/cdeep3m/mito_testsample/testset/images.044.png
Padding images
.
.
.
Saving: /home/ubuntu/mito_testaugimages/Pkg001_Z01/test_data_full_stacks_v16.h5
Elapsed time is 1.75296 seconds.
Image Augmentation completed
Created 1 packages in x/y with 1 z-stacks
Data stored in:
 /home/ubuntu/mito_testaugimages

Step 3) Run runprediction.sh

This step runs caffe prediction writing results to ~/predictout30k directory and takes about 3-5 minutes.

runprediction.sh ~/train_out/ ~/mito_testaugimages/ ~/predictout30k

Output:

octave: X11 DISPLAY environment variable not set
octave: disabling GUI features
Verifying input training data is valid ... skipping check, TODO need to fix this.
Verifying image data and getting Pkg folders ... skipping check, TODO need to fix this.
Creating output directories and creating run scripts ... success
A new directory has been created: /home/ubuntu/predictout30k
In this directory are 3 directories 1fm,3fm,5fm which
will contain the results from running prediction with caffe
There are also two scripts:
caffe_predict.sh -- Runs caffe prediction single model
run_all_predict.sh -- Runs caffe_predict.sh serially for all 3 models
Running Prediction
Trained Model Dir: /home/ubuntu/sbem/mitochrondria/xy5.9nm40nmz/30000iterations_train_out
Image Dir: /home/ubuntu/mito_testaugimages/
Running 1fm predict 1 package(s) to process
  Processing Pkg001_Z01 1 of 1 ................
Running StartPostprocessing.m /home/ubuntu/predictout30k/1fm/Pkg001_Z01
Running Merge_LargeData.m /home/ubuntu/predictout30k/1fm
real 140.78
user 52.74
sys 26.08
Running 3fm predict 1 package(s) to process
  Processing Pkg001_Z01 1 of 1 ................
Running StartPostprocessing.m /home/ubuntu/predictout30k/3fm/Pkg001_Z01
Running Merge_LargeData.m /home/ubuntu/predictout30k/3fm
real 95.75
user 66.90
sys 27.04
Running 5fm predict 1 package(s) to process
  Processing Pkg001_Z01 1 of 1 ................
Running StartPostprocessing.m /home/ubuntu/predictout30k/5fm/Pkg001_Z01
Running Merge_LargeData.m /home/ubuntu/predictout30k/5fm
real 97.15
user 67.03
sys 26.78
Prediction has completed. Have a nice day!
Prediction has completed. Results are stored in /home/ubuntu/predictout30k
Have a nice day!

Step 4) Run EnsemblePredictions.m

Merges segmentations from 1fm, 3fm, and 5fm models. Final segmentations will be put in directory specified by last argument to EnsemblePredictions.m in case below its ~/predictout30k/ensembled which will contain a set of 8-bit grayscale png files where pixel intensity denotes likelihood of feature that model was trained to detect. This step only takes a few seconds to run.

EnsemblePredictions.m ~/predictout30k/1fm ~/predictout30k/3fm ~/predictout30k/5fm ~/predictout30k/ensembled

Output:

octave: X11 DISPLAY environment variable not set
octave: disabling GUI features
warning: your version of GraphicsMagick limits images to 16 bits per pixel
warning: called from
    imformats>default_formats at line 256 column 11
    imformats at line 79 column 3
    imageIO at line 106 column 11
    imread at line 106 column 30
    /home/ubuntu/cdeep3m/EnsemblePredictions.m at line 50 column 31
Saving Image # 1 of 5: /home/ubuntu/predictout30k/ensembled/Segmented_0001.png
Saving Image # 2 of 5: /home/ubuntu/predictout30k/ensembled/Segmented_0002.png
Saving Image # 3 of 5: /home/ubuntu/predictout30k/ensembled/Segmented_0003.png
Saving Image # 4 of 5: /home/ubuntu/predictout30k/ensembled/Segmented_0004.png
Saving Image # 5 of 5: /home/ubuntu/predictout30k/ensembled/Segmented_0005.png
Elapsed time for merging predictions is 000002 seconds.

The final results can be seen when running ls

ls ~/predictout30k/ensembled

Output:

Segmented_0001.png  Segmented_0003.png  Segmented_0005.png
Segmented_0002.png  Segmented_0004.png