Ch/add quarry ex

.gitignore

@@ -1,4 +1,5 @@
 output*
+DEM*
 cache
 
 # Byte-compiled / optimized / DLL files

examples/quarry-detection/README.md

@@ -0,0 +1,41 @@
+# Example: detection of quarries
+
+A sample working setup is provided here, enabling the end-user to detect quarries and mineral extraction sites in Switzerland over several years. <br>
+It consists of the following elements:
+
+- the read-to-use configuration files:
+    - `config_trne.yaml`,
+    - `config_prd.yaml`,
+    - `detectron2_config_dqry.yaml`,
+- the input data in the `data` subfolder:
+    - quarries shapefile from the product [swissTLM3D](https://www.swisstopo.admin.ch/fr/geodata/landscape/tlm3d.html), revised and synchronised with the 2020 [SWISSIMAGE](https://www.swisstopo.admin.ch/fr/geodata/images/ortho/swissimage10.html) mosaic (**label**),
+    - the delimitation of the AOI to perform inference predictions (**AOI**),
+    - the swiss DEM raster is too large to be saved on this platform but can be downloaded from this [link](https://github.com/lukasmartinelli/swissdem) using the EPSG:4326 - WGS 84 coordinate reference system. The raster must be first reprojected to EPSG:2056 - CH1903+ / LV95, named `switzerland_dem_EPSG2056.tif`and located in the **DEM** subfolder.
+- a data preparation script (`prepare_data.py`) producing the files to be used as input to the `generate_tilesets.py`script.
+- a results post-processing script (`filter_prediction.py`) filtering the predictions, produced from `make_prediction.py`script, to the final shapefile 
+
+After creating and a new environment in python 3.8, the end-to-end workflow can be run by issuing the following list of commands, straight from this folder:
+
+```bash
+$ sudo apt-get install -y python3-gdal gdal-bin libgdal-dev gcc g++ python3.8-dev
+$ pip install -r ../../requirements.txt
+$ python3 prepare_data.py config_trne.yaml
+$ python3 ../../scripts/generate_tilesets.py config_trne.yaml
+$ python3 ../../scripts/train_model.py config_trne.yaml
+$ python3 ../../scripts/make_predictions.py config_trne.yaml
+$ python3 ../../scripts/assess_predictions.py config_trne.yaml
+$ python3 prepare_data.py config_prd.yaml
+$ python3 ../../scripts/generate_tilesets.py config_prd.yaml
+$ python3 ../../scripts/make_predictions.py config_prd.yaml
+$ python3 filter_detection.py config_prd.yaml
+```
+
+We strongly encourage the end-user to review the provided `config_trne.yaml` and `config_prd.yaml` files as well as the various output files, a list of which is printed by each script before exiting.
+
+The model is trained on the 2020 [SWISSIMAGE](https://www.swisstopo.admin.ch/fr/geodata/images/ortho/swissimage10.html) mosaic. Inference can be performed on SWISSIMAGE mosaics of the product SWISSIMAGE time travel by changing the year in `config_prd.yaml`. It should be noted that the model has been trained on RGB color images and might not perform as well on Black and White images.
+
+For more information about this project, you can consult [the associated repository](https://github.com/swiss-territorial-data-lab/proj-dqry) (not public yet).
+
+## Disclaimer
+
+Depending on the end purpose, we strongly recommend users not to take for granted the detections obtained through this code. Indeed, results can exhibit false positives and false negatives, as is the case in all Machine Learning-based approaches.

examples/quarry-detection/config_prd.yaml

@@ -0,0 +1,64 @@
+###################################
+####### Inference detection ####### 
+# Automatic detection of Quarries and Mineral Extraction Sites (MES) in images
+
+# 1-Produce tiles geometry according to the AOI extent and zoom level
+prepare_data.py: 
+  srs: "EPSG:2056"                       # Projection of the input file
+  datasets:
+    labels_shapefile: ./data/AOI/AOI_2020.shp
+  output_folder: ./output/output-prd
+  zoom_level: 16   
+
+# 2-Fetch of tiles (online server) and split into 3 datasets: train, test, validation
+generate_tilesets.py:
+  debug_mode: False # sample of tiles
+  datasets:
+    aoi_tiles_geojson: ./output/output-prd/tiles.geojson
+    orthophotos_web_service:
+      type: XYZ # supported values: 1. MIL = Map Image Layer 2. WMS 3. XYZ
+      url: https://wmts.geo.admin.ch/1.0.0/ch.swisstopo.swissimage-product/default/2020/3857/{z}/{x}/{y}.jpeg
+  output_folder: ./output/output-prd
+  tile_size: 256 # per side, in pixels
+  overwrite: True
+  n_jobs: 10
+  COCO_metadata:
+    year: 2021
+    version: 1.0
+    description: Swiss Image Hinterground w/ Quarries and Mineral Exploitation Sites detection
+    contributor: swisstopo
+    url: https://swisstopo.ch
+    license:
+      name: Unknown
+      url:
+    category:
+      name: "Quarry"
+      supercategory: "Land usage"
+
+# 3-Perform the object detection based on the optimized trained model
+make_predictions.py:
+  working_folder: ./output/output-prd
+  log_subfolder: logs
+  sample_tagged_img_subfolder: sample_tagged_images
+  COCO_files: # relative paths, w/ respect to the working_folder
+    oth: COCO_oth.json
+  detectron2_config_file: '../../detectron2_config_dqry.yaml' # path relative to the working_folder
+  model_weights:
+    pth_file: '../output-trne/logs/model_0002999.pth' # trained model minimizing the validation loss curve, monitor the training process via tensorboard (tensorboard --logdir </logs>) 
+  image_metadata_json: './output/output-prd/img_metadata.json'
+  rdp_simplification: # rdp = Ramer-Douglas-Peucker
+    enabled: True
+    epsilon: 2.0 # cf. https://rdp.readthedocs.io/en/latest/
+  score_lower_threshold: 0.3
+
+# 4-Filtering and merging prediction polygons to improve results
+filter_prediction.py:
+  year: 2020
+  input: ./output/output-prd/oth_predictions_at_0dot3_threshold.gpkg
+  labels_shapefile: ./data/AOI/AOI_2020.shp
+  dem: ./data/DEM/switzerland_dem_EPSG2056.tif
+  elevation: 1200.0 # m, altitude threshold
+  score: 0.95 # prediction score (from 0 to 1) provided by detectron2
+  distance: 10 # m, distance use as a buffer to merge close polygons (likely to belong to the same object) together
+  area: 5000.0 # m2, area threshold under which polygons are discarded
+  output: ./output/output-prd/oth_prediction_at_0dot3_threshold_year-{year}_score-{score}_area-{area}_elevation-{elevation}_distance-{distance}.geojson

examples/quarry-detection/config_trne.yaml

@@ -0,0 +1,80 @@
+#############################################
+####### Model training and evaluation ####### 
+# Training of automatic detection of Quarries and Mineral Extraction Sites (MES) in images with a provided ground truth
+
+# 1-Produce tiles geometry according to the AOI extent and zoom level
+prepare_data.py:  
+  srs: "EPSG:2056"
+  datasets:
+    labels_shapefile: ./data/labels/tlm-hr-trn-topo.shp
+  output_folder: ./output/output-trne
+  zoom_level: 16 
+
+# 2-Fetch of tiles (online server) and split into 3 datasets: train, test, validation
+generate_tilesets.py:
+  debug_mode: False # sample of tiles
+  datasets:
+    aoi_tiles_geojson: ./output/output-trne/tiles.geojson
+    ground_truth_labels_geojson: ./output/output-trne/labels.geojson
+    orthophotos_web_service:
+      type: XYZ # supported values: 1. MIL = Map Image Layer 2. WMS 3. XYZ
+      url: https://wmts.geo.admin.ch/1.0.0/ch.swisstopo.swissimage-product/default/2020/3857/{z}/{x}/{y}.jpeg
+  output_folder: ./output/output-trne
+  tile_size: 256 # per side, in pixels
+  overwrite: False
+  n_jobs: 10
+  COCO_metadata:
+    year: 2021
+    version: 1.0
+    description: Swiss Image Hinterground w/ Quarries and Mineral Exploitation Sites detection
+    contributor: swisstopo
+    url: https://swisstopo.ch
+    license:
+      name: Unknown
+      url:
+    category:
+      name: "Quarry"
+      supercategory: "Land usage"
+
+# 3-Train the model with the detectron2 algorithm
+train_model.py:
+  working_folder: ./output/output-trne
+  log_subfolder: logs
+  sample_tagged_img_subfolder: sample_tagged_images
+  COCO_files: # relative paths, w/ respect to the working_folder
+    trn: COCO_trn.json
+    val: COCO_val.json
+    tst: COCO_tst.json
+  detectron2_config_file: '../../detectron2_config_dqry.yaml' # path relative to the working_folder
+  model_weights:
+    model_zoo_checkpoint_url: "COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_1x.yaml"
+
+# 4-Perform the object detection based on the optimized trained model
+make_predictions.py:
+  working_folder: ./output/output-trne
+  log_subfolder: logs
+  sample_tagged_img_subfolder: sample_tagged_images
+  COCO_files: # relative paths, w/ respect to the working_folder
+    trn: COCO_trn.json
+    val: COCO_val.json
+    tst: COCO_tst.json
+  detectron2_config_file: '../../detectron2_config_dqry.yaml' # path relative to the working_folder
+  model_weights:
+    pth_file: './logs/model_0002999.pth' # trained model minimizing the validation loss curve, monitor the training process via tensorboard (tensorboard --logdir </logs>)
+  image_metadata_json: './output/output-trne/img_metadata.json'
+  rdp_simplification: # rdp = Ramer-Douglas-Peucker
+    enabled: true
+    epsilon: 2.0 # cf. https://rdp.readthedocs.io/en/latest/
+  score_lower_threshold: 0.05
+
+# 5-Evaluate the quality of the prediction for the different datasets with metrics calculation
+assess_predictions.py:
+  datasets:
+    ground_truth_labels_geojson: ./output/output-trne/labels.geojson
+    image_metadata_json: ./output/output-trne/img_metadata.json
+    split_aoi_tiles_geojson: ./output/output-trne/split_aoi_tiles.geojson # aoi = Area of Interest
+    predictions:
+      trn: ./output/output-trne/trn_predictions_at_0dot05_threshold.gpkg
+      val: ./output/output-trne/val_predictions_at_0dot05_threshold.gpkg
+      tst: ./output/output-trne/tst_predictions_at_0dot05_threshold.gpkg
+  output_folder: ./output/output-trne

examples/quarry-detection/data/AOI/AOI_2020.cpg

@@ -0,0 +1 @@
+UTF-8

examples/quarry-detection/data/AOI/AOI_2020.prj

@@ -0,0 +1 @@
+PROJCS["CH1903+_LV95",GEOGCS["GCS_CH1903+",DATUM["D_CH1903+",SPHEROID["Bessel_1841",6377397.155,299.1528128]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Hotine_Oblique_Mercator_Azimuth_Center"],PARAMETER["False_Easting",2600000.0],PARAMETER["False_Northing",1200000.0],PARAMETER["Scale_Factor",1.0],PARAMETER["Azimuth",90.0],PARAMETER["Longitude_Of_Center",7.43958333333333],PARAMETER["Latitude_Of_Center",46.9524055555556],UNIT["Meter",1.0]]

examples/quarry-detection/data/labels/tlm-hr-trn-topo.prj

@@ -0,0 +1 @@
+PROJCS["CH1903+_LV95",GEOGCS["GCS_CH1903+",DATUM["D_CH1903+",SPHEROID["Bessel_1841",6377397.155,299.1528128]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Hotine_Oblique_Mercator_Azimuth_Center"],PARAMETER["False_Easting",2600000.0],PARAMETER["False_Northing",1200000.0],PARAMETER["Scale_Factor",1.0],PARAMETER["Azimuth",90.0],PARAMETER["Longitude_Of_Center",7.43958333333333],PARAMETER["Latitude_Of_Center",46.9524055555556],UNIT["Meter",1.0]]