Add motivation to README #71

stijnvermeeren-swisstopo · 2024-08-12T13:26:59Z

No description provided.

github-actions · 2024-08-12T13:27:54Z

Coverage Report

File	Stmts	Miss	Cover	Missing
src/stratigraphy
__init__.py	8	1	88%	11
extract.py	188	188	0%	3–482
get_files.py	19	19	0%	3–47
line_detection.py	26	26	0%	3–76
main.py	95	95	0%	3–237
src/stratigraphy/util
boundarydepthcolumnvalidator.py	41	20	51%	47, 57, 60, 81–84, 109–127, 139–148
coordinate_extraction.py	127	7	94%	31, 62, 75–76, 80, 205, 328
dataclasses.py	32	3	91%	37–39
depthcolumn.py	194	64	67%	26, 30, 51, 57, 60–61, 85, 88, 95, 102, 110–111, 121, 138–154, 192, 229, 248–256, 267, 272, 279, 310, 315–322, 337–338, 381–423
depthcolumnentry.py	26	7	73%	12, 15, 29–30, 33, 45, 52
description_block_splitter.py	70	2	97%	24, 139
draw.py	80	80	0%	3–244
duplicate_detection.py	51	51	0%	3–146
extract_text.py	27	2	93%	38–39
find_depth_columns.py	91	6	93%	42–43, 71, 83, 176–177
find_description.py	63	28	56%	27–35, 50–63, 79–95, 172–175
geometric_line_utilities.py	86	2	98%	82, 132
interval.py	106	55	48%	24–27, 31–34, 39, 44, 47, 57–59, 99–145, 166, 171–187
language_detection.py	18	18	0%	3–45
layer_identifier_column.py	91	91	0%	3–227
line.py	49	4	92%	25, 42, 51, 98
linesquadtree.py	46	1	98%	76
plot_utils.py	43	43	0%	3–120
predictions.py	130	130	0%	3–272
textblock.py	74	8	89%	27, 51, 63, 75, 98, 119, 127, 155
util.py	40	18	55%	22, 40–47, 61–63, 87–88, 100–105
TOTAL	1821	969	47%

Tests	Skipped	Failures	Errors	Time
61	0 💤	0 ❌	0 🔥	0.979s ⏱️

dcleres · 2024-08-12T13:36:06Z

README.md


-Note that the project is under active development and there is no release to this date, nor has the project reached a maturity such that it could be used.
+The Federal Office of Topography swisstopo is Switzerland's geoinformation centre. The Swiss Geological Survey at swisstopo is the federal competence centre for the collection, analysis, storage and provision of geological data of national interest. 


Missing comma:

The Federal Office of Topography swisstopo is Switzerland's geoinformation centre. The Swiss Geological Survey at swisstopo is the federal competence centre for the collection, analysis, storage, and provision of geological data of national interest.

That's not a universally accepted rule :) -> https://en.wikipedia.org/wiki/Serial_comma
However, I don't feel very strongly about it, and I can add the comma.

A general note: could you try using the "add a suggestion" Github feature for suggesting minor changes like this one? That would make it easier to find and to accept the suggested changes.

dcleres · 2024-08-12T13:46:09Z

Thank you for improving the README.md. It is much clearer now.

Do you think we should also reference the two related label studio forks?

dcleres

It looks good to me. The content is good and very clear. I spotted a few typos that you could address before merging. Thank you for clarifying this part of the project!

dcleres · 2024-08-12T13:38:45Z

README.md

-Only German and French borehole profiles are supported as of now.
+Therefore, the goal of this project is to automate the extraction of structured data from borehole profiles as much as possible. As far as swisstopo is concerned, the use case is to integrate the data extraction pipeline with the application boreholes.swissgeol.ch ([Github](https://github.com/swisstopo/swissgeol-boreholes-suite)), where a user interface for efficient quality control of the automatically extracted data will also be implemented.
+
+All code and documentation is published in this Github repository as open source software. All other persons, companies or agencies who manage borehole data of their own, are welcome to use the data extraction pipeline on their own data and to contribute to the project with their own improvements/additions.


Github -> GitHub

dcleres · 2024-08-12T13:40:03Z

README.md

+* Lithology / stratigraphy
+  * **Depths** (upper and lower bound of each layer)
+  * **Material descriptions** (as plain text)
+  * USCS classification, color, consistency, plasticity...


Why not italic or bold?

Not being actively worked on (at least not planned for the rest of the year). Same for the others.

dcleres · 2024-08-12T13:40:07Z

README.md

+  * **Depths** (upper and lower bound of each layer)
+  * **Material descriptions** (as plain text)
+  * USCS classification, color, consistency, plasticity...
+  * Geological interpretations


Why not italic or bold?

dcleres · 2024-08-12T13:40:12Z

README.md

+  * Geological interpretations
+* Other
+  * _Hydrogeology (ground water levels)_
+  * Instrumentation


Why not italic or bold?

dcleres · 2024-08-12T13:40:17Z

README.md

+* Other
+  * _Hydrogeology (ground water levels)_
+  * Instrumentation
+  * Casing


Why not italic or bold?

dcleres · 2024-08-12T13:40:22Z

README.md

+  * _Hydrogeology (ground water levels)_
+  * Instrumentation
+  * Casing
+  * Borehole geometry


Why not italic or bold?

dcleres · 2024-08-12T13:41:20Z

README.md

+
+### Related work
+
+Existing work related to this project is mostly focussed on the extraction and classification of specific properties from textual geological descriptions.  Notable examples include [GEOBERTje](https://www.arxiv.org/abs/2407.10991) (Belgium), [geo-ner-model](https://github.com/BritishGeologicalSurvey/geo-ner-model) (UK), [GeoVec](https://www.sciencedirect.com/science/article/pii/S0098300419306533) und [dh2loop](https://github.com/Loop3D/dh2loop) (Australia). The scope of this project is considerable wider, in particular regarding the goal of understanding borehole profiles in various languages and with an unknown layout, where the document structure first needs to be understood, before the relevant text fragments can be identified and extracted.


Two spaces: descriptions. Notable

dcleres · 2024-08-12T13:43:00Z

README.md

+
+Existing work related to this project is mostly focussed on the extraction and classification of specific properties from textual geological descriptions.  Notable examples include [GEOBERTje](https://www.arxiv.org/abs/2407.10991) (Belgium), [geo-ner-model](https://github.com/BritishGeologicalSurvey/geo-ner-model) (UK), [GeoVec](https://www.sciencedirect.com/science/article/pii/S0098300419306533) und [dh2loop](https://github.com/Loop3D/dh2loop) (Australia). The scope of this project is considerable wider, in particular regarding the goal of understanding borehole profiles in various languages and with an unknown layout, where the document structure first needs to be understood, before the relevant text fragments can be identified and extracted.
+
+The automatic data extraction pipeline can be considered to belong to the field or [automatic/intelligent document processing](https://en.wikipedia.org/wiki/Document_processing). As such, it involves a combination of methods from multiple fields in data science and machine learning, in particular computer vision (e.g. object detection, line detection) and natural language processing (large language models, named entity recognition). Some of these have already been implemented (e.g. the [Line Segment Detector](https://docs.opencv.org/3.4/db/d73/classcv_1_1LineSegmentDetector.html) algorithm), others are planned as future work.


belong to the field of

stijnvermeeren-swisstopo · 2024-08-12T15:57:06Z

Do you think we should also reference the two related label studio forks?

Unless we decide that we actually want to maintain those forks and put them into production after all, I would not explicitly mention them here.

add motivation to README

c94edaf

dcleres reviewed Aug 12, 2024

View reviewed changes

dcleres approved these changes Aug 12, 2024

View reviewed changes

stijnvermeeren-swisstopo added 2 commits August 12, 2024 18:05

code review

42edc1a

clarify legend for extracted properties

eb81399

stijnvermeeren-swisstopo merged commit a2ae1cb into main Aug 13, 2024
3 checks passed

stijnvermeeren-swisstopo deleted the readme-additions branch August 13, 2024 11:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add motivation to README #71

Add motivation to README #71

stijnvermeeren-swisstopo commented Aug 12, 2024

github-actions bot commented Aug 12, 2024 •

edited

Loading

dcleres Aug 12, 2024

stijnvermeeren-swisstopo Aug 12, 2024

dcleres commented Aug 12, 2024

dcleres left a comment

dcleres Aug 12, 2024

dcleres Aug 12, 2024

stijnvermeeren-swisstopo Aug 12, 2024

dcleres Aug 12, 2024

dcleres Aug 12, 2024

dcleres Aug 12, 2024

dcleres Aug 12, 2024

dcleres Aug 12, 2024

dcleres Aug 12, 2024

stijnvermeeren-swisstopo commented Aug 12, 2024


		Note that the project is under active development and there is no release to this date, nor has the project reached a maturity such that it could be used.
		The Federal Office of Topography swisstopo is Switzerland's geoinformation centre. The Swiss Geological Survey at swisstopo is the federal competence centre for the collection, analysis, storage and provision of geological data of national interest.


		### Related work

		Existing work related to this project is mostly focussed on the extraction and classification of specific properties from textual geological descriptions. Notable examples include [GEOBERTje](https://www.arxiv.org/abs/2407.10991) (Belgium), [geo-ner-model](https://github.com/BritishGeologicalSurvey/geo-ner-model) (UK), [GeoVec](https://www.sciencedirect.com/science/article/pii/S0098300419306533) und [dh2loop](https://github.com/Loop3D/dh2loop) (Australia). The scope of this project is considerable wider, in particular regarding the goal of understanding borehole profiles in various languages and with an unknown layout, where the document structure first needs to be understood, before the relevant text fragments can be identified and extracted.


		Existing work related to this project is mostly focussed on the extraction and classification of specific properties from textual geological descriptions. Notable examples include [GEOBERTje](https://www.arxiv.org/abs/2407.10991) (Belgium), [geo-ner-model](https://github.com/BritishGeologicalSurvey/geo-ner-model) (UK), [GeoVec](https://www.sciencedirect.com/science/article/pii/S0098300419306533) und [dh2loop](https://github.com/Loop3D/dh2loop) (Australia). The scope of this project is considerable wider, in particular regarding the goal of understanding borehole profiles in various languages and with an unknown layout, where the document structure first needs to be understood, before the relevant text fragments can be identified and extracted.

		The automatic data extraction pipeline can be considered to belong to the field or [automatic/intelligent document processing](https://en.wikipedia.org/wiki/Document_processing). As such, it involves a combination of methods from multiple fields in data science and machine learning, in particular computer vision (e.g. object detection, line detection) and natural language processing (large language models, named entity recognition). Some of these have already been implemented (e.g. the [Line Segment Detector](https://docs.opencv.org/3.4/db/d73/classcv_1_1LineSegmentDetector.html) algorithm), others are planned as future work.

Add motivation to README #71

Add motivation to README #71

Conversation

stijnvermeeren-swisstopo commented Aug 12, 2024

github-actions bot commented Aug 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dcleres commented Aug 12, 2024

dcleres left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stijnvermeeren-swisstopo commented Aug 12, 2024

github-actions bot commented Aug 12, 2024 •

edited

Loading