Merge pull request #1 from tcolligan4/tommy

Tommy

.gitignore

@@ -8,6 +8,7 @@ __pycache__/
 
 # Distribution / packaging
 .Python
+.condaauto
 env/
 build/
 develop-eggs/
@@ -110,4 +111,31 @@ scene_list/
 spatial_data/MT
 tests/data
 gee_training.py
-model_data/
+model_data/
+fully-conv-classification/*shp
+fully-conv-classification/*dbf
+fully-conv-classification/*cpg
+fully-conv-classification/*prj
+fully-conv-classification/*sh
+fully-conv-classification/*tif
+fully-conv-classification/*txt
+fully-conv-classification/*shx
+fully-conv-classification/compare_model_outputs/
+fully-conv-classification/binary/
+fully-conv-classification/all_models/
+fully-conv-classification/models/
+fully-conv-classification/copy_files.py
+fully-conv-classification/examine_dates.py
+fully-conv-classification/evaluated_over_mt/
+fully-conv-classification/evaluated_images/
+fully-conv-classification/no_expit/
+fully-conv-classification/tags
+fully-conv-classification/random_tifs/
+fully-conv-classification/random_majority_files/
+fully-conv-classification/model_ensemble.py
+fully-conv-classification/hyperparameters/
+fully-conv-classification/mean_of_three/
+fully-conv-classification/ensemble_models/
+fully-conv-classification/precision_and_recall_multiple_files.py
+fully-conv-classification/plot_cmat.py
+fully-conv-classification/shapefile_data/

README.md

@@ -8,7 +8,7 @@ This package is a little difficult to create a working python interpreter for.
 First, get [Anaconda](anaconda.org) and [git](https://git-scm.com/), these tools
 are important here.
 
-Next, create your environment
+Next, create your environment.
 
 ``` conda create -n irri python=3.6```
 
@@ -18,4 +18,50 @@ Then get the latest gdal:
 
 Then the latest master branch of rasterio:
 
-```pip install git+https://github.com/mapbox/rasterio.git```
+```pip install git+https://github.com/mapbox/rasterio.git```
+
+Install Metio:
+
+```pip install git+https://github.com/tcolligan4/Metio.git```
+
+Install SatelliteImage:
+
+```pip install git+https://github.com/dgketchum/satellite_image.git```
+
+# usage
+## 1 
+Edit the file ```runspec.py``` and fill out the two methods ```assign_shapefile_class_code``` and ```assign_shapefile_year```. These functions take as input a path to a shapefile and return integers corresponding to the class code of the vector data in the shapefile and the year the data was recorded, respectively. This means shapefiles should be split up according to class code and year before the next step.
+
+Also in ``runspec.py``, select the bands you want by editing ``landsat_rasters()``, ``static_rasters()``, and ``climate_rasters()``. ``mask_rasters()`` specifies water and cloud masks. 
+## 2
+use split_shapefile.py to split the training data shapefiles into WRS2 descending path/rows. Ex:
+```
+python split_shapefile.py --shapefile-dir /home/thomas/training_data/ --output-dir /home/thomas/split_training_data/
+```
+Default extension is .shp for the input shapefiles.
+## 3
+run extract_training_data.py to extract training data. This relies on the methods ``assign_shapefile_class_code`` and ``assign_shapefile_year`` that reside in ``runspec.py``. It also downloads all image data to image-dir. 
+```
+python extract_training_data.py --shapefile-dir /home/thomas/split_training_data --image-dir /home/thomas/landsat/ --training-dir /home/thomas/irrmapper/data/train/ --n-classes 5
+```
+Before running this, check ```_one_hot_from_labels_mc()``` in ```extract_training_data.py```. This applies a border class to shapefile data of class code 0 for reasons related to mapping irrigation. If this is not what you want, comment out the conditional in this function.  
+
+## 4 
+train a model with train_model.py.
+
+# TODO: 
+Make training a model easier [x] (i.e. don't require a separate weights matrix and stop computing softmax within the network)
+
+Implement binary dice loss [x]
+
+Get rid of class_7_data necessity 
+
+Implement multiclass dice loss
+
+add binary classification possibility [x]
+
+add raster training data extraction
+
+Make the images downloaded pull from runspec.py, not automatically download all possible bands
+
+add paths_map_single_scene flag to extract_training_data

fully-conv-classification/_clip_rasters.py

@@ -0,0 +1,20 @@
+import os
+from argparse import ArgumentParser
+from data_utils import clip_raster
+from glob import glob
+
+def _parse_path_row(f):
+    bs = os.path.basename(f).split("_")
+    return bs[0], bs[1]
+
+if __name__ == '__main__':
+
+    ap = ArgumentParser()
+    ap.add_argument('--raster', type=str, required=True)
+    ap.add_argument('--out-dir', type=str, required=True)
+    ap.add_argument('--outfile', type=str)
+    args = ap.parse_args()
+    if args.outfile is None:
+        outfile = args.raster
+    path, row = _parse_path_row(args.raster)
+    clip_raster(args.raster, int(path), int(row), outfile=outfile)