Feat: Material Block recognition for Geneva Layout #45

redur · 2024-05-21T07:56:31Z

Includes functionalities to recognize material description blocks for the Geneva borehole profile layout. The code detects so called layer index columns which indicate the presence of a material description rect as well as their position marks how to split the material description rect into material blocks. Note: Some visualization code is part of this commit which should be dropped in a follow up commit.

redur · 2024-05-21T07:58:00Z

config/matching_params.yml

  fr:
    including_expressions:
      - sol
-      - végétal
+      - végétal  # remove accents generally; ocr might be wrong
      - dallage
      - terre
      - bitume
      - bitumineux
-      - grave d'infrastructure
+      - grave d'infrastructure  # what happens if we remove this?
      - sable
      - limon
      - gravier
      - asphalte
-      - humus
+      - humus  # hummus maybe?
      - brun
      - gris
+      - grise
      - mou
      - dur
+      - dure
+      - ferme
      - racine
      - revetement
      - pierre
      - beige
      - beton
      - craie
      - marne
-      - materiau de base
+      - materiau
      - matrice sableuse
-      - enrobé
+      - enrobé  # accent --> check what happens if it's removed
+      - terrain
+      - remblais
+      - remblai
+      - molasse
+      - phase
+      - formations
+      - limoneuse
+      - argileuse
+      - argileux
+      - mousse
    excluding_expressions:
      - monsieur
      - fin


I checked our expressions with a native French speaker who also has experience with NLP with the French language. We looked at borehole profiles together and extended the list. Also, he suggested to remove all accents because OCR frequently makes errors there.

That's the reasons for the changes and comments here.

github-actions · 2024-05-21T07:58:03Z

Coverage Report

File	Stmts	Miss	Cover	Missing
src/stratigraphy
__init__.py	8	1	88%	11
extract.py	214	214	0%	3–522
get_files.py	21	21	0%	3–48
line_detection.py	26	26	0%	3–76
main.py	91	91	0%	3–232
src/stratigraphy/util
coordinate_extraction.py	128	31	76%	30, 50, 54, 58–66, 143, 163, 235–241, 250–252, 268–282
dataclasses.py	32	3	91%	37–39
depthcolumn.py	206	67	67%	26, 30, 51, 57, 60–61, 85, 88, 95, 102, 110–111, 121, 138–154, 199, 238, 254–262, 274, 279, 286, 310, 314, 343, 364, 367–378, 393–394, 439–481
depthcolumnentry.py	20	4	80%	12, 15, 27, 34
description_block_splitter.py	70	2	97%	24, 139
draw.py	73	73	0%	3–225
duplicate_detection.py	32	32	0%	3–81
find_depth_columns.py	82	4	95%	56–57, 151–152
find_description.py	63	28	56%	27–35, 50–63, 79–95, 172–175
geometric_line_utilities.py	87	2	98%	83, 133
interval.py	107	52	51%	25–28, 32–35, 40, 45, 48, 100–146, 167, 172–188
language_detection.py	18	18	0%	3–43
layer_identifier_column.py	61	61	0%	3–162
line.py	49	26	47%	25, 42, 51, 65–95, 98
linesquadtree.py	46	1	98%	76
plot_utils.py	44	44	0%	3–121
predictions.py	187	187	0%	3–385
textblock.py	74	8	89%	27, 51, 63, 75, 98, 119, 127, 155
util.py	40	22	45%	15–18, 22, 26, 40–47, 61–63, 87–88, 100–105
TOTAL	1779	1018	43%

Tests	Skipped	Failures	Errors	Time
58	0 💤	0 ❌	0 🔥	0.590s ⏱️

redur · 2024-05-21T07:59:10Z

src/stratigraphy/extract.py

+            # Visualization: To be dropped before merging to main.
+            for layer_index_column in layer_index_columns:
+                fitz.utils.draw_rect(
+                    page, layer_index_column.rect() * page.derotation_matrix, color=fitz.utils.getColor("blue")
+                )
+            for block in blocks:
+                fitz.utils.draw_rect(page, block.rect * page.derotation_matrix, color=fitz.utils.getColor("red"))
+            fitz.utils.draw_rect(
+                page, material_description_rect * page.derotation_matrix, color=fitz.utils.getColor("blue")
+            )
+            page.parent.save(DATAPATH / "_temp" / "output.pdf", garbage=4, deflate=True, clean=True)


This will need to be dropped. Is useful to inspect the recognized layer index columns as well as the material description rect.

redur · 2024-05-21T08:00:59Z

src/stratigraphy/extract.py

+            )
+            page.parent.save(DATAPATH / "_temp" / "output.pdf", garbage=4, deflate=True, clean=True)
+
+            return predictions, json_filtered_pairs


For now, I just exit here if there's a layer index column.

Some thoughts:
I believe it should be possible to consider a layer index column a special case of a depth column, and continue the "normal" way. I believe this will be "cleaner" to move on.

redur · 2024-05-21T08:01:52Z

src/stratigraphy/util/layer_index_column.py

+from stratigraphy.util.line import TextLine
+
+
+class LayerIndexColumn:


Already now, this is quite similar to a depth column.

redur · 2024-05-21T08:02:22Z

src/stratigraphy/util/find_description.py

+    return blocks
+
+
+def matching_blocks(all_lines: list[TextLine], line_index: int, next_layer_index: TextLine | None) -> list[TextBlock]:


Could become a method of LayerIndexColumn.

If next_layer_index is not just an index but of type TextLine | None , then probably next_layer is a better name?

I did not create a class object LayerIndexEntry similar to the entries of depthcolumns. The entries of the LayerIndex are simply TextLine objects.

What I mean here is the next layer index "entry". Therefore I called it next_layer_index. Does that make sense?

Moving forward and considering LayerIndexColumn as a special case of a DepthColum we might want to create the "LayerIndexEntry" object as well, and then this will be clearer. What do you think?

stijnvermeeren-swisstopo

I think that this is already a pretty decent first implementation, though obviously there is room for some simplification of the code (sharing more code between the "index column" and "depth column" implementations.

I would propose to merge this (after fixing the minor comments) and then to improve the code in a next iteration.

src/stratigraphy/util/layer_index_column.py

stijnvermeeren-swisstopo · 2024-05-21T08:23:52Z

src/stratigraphy/util/find_description.py

+    return blocks
+
+
+def matching_blocks(all_lines: list[TextLine], line_index: int, next_layer_index: TextLine | None) -> list[TextBlock]:


If next_layer_index is not just an index but of type TextLine | None , then probably next_layer is a better name?

src/stratigraphy/util/find_description.py

src/stratigraphy/util/layer_index_column.py

stijnvermeeren-swisstopo · 2024-05-21T08:55:01Z

src/stratigraphy/extract.py

+            )
+            page.parent.save(DATAPATH / "_temp" / "output.pdf", garbage=4, deflate=True, clean=True)
+
+            return predictions, json_filtered_pairs


redur commented May 21, 2024

View reviewed changes

redur requested a review from stijnvermeeren-swisstopo May 21, 2024 08:02

stijnvermeeren-swisstopo requested changes May 21, 2024

View reviewed changes

redur added 2 commits May 21, 2024 12:34

Address PR comments.

7c161d6

Rename LayerIndexColumn to LayerIdentifierColumn

1a60ddc

redur requested a review from stijnvermeeren-swisstopo May 21, 2024 11:47

improve comments + use TextWord instead of TextLine where appropriate

d4dac85

stijnvermeeren-swisstopo approved these changes May 22, 2024

View reviewed changes

redur self-assigned this May 22, 2024

redur merged commit e156378 into main May 22, 2024
3 checks passed

redur deleted the feat/improve_geneva_layout branch May 22, 2024 09:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat: Material Block recognition for Geneva Layout #45

Feat: Material Block recognition for Geneva Layout #45

redur commented May 21, 2024

redur May 21, 2024 •

edited

Loading

github-actions bot commented May 21, 2024 •

edited

Loading

redur May 21, 2024

redur May 21, 2024

stijnvermeeren-swisstopo May 21, 2024

redur May 21, 2024

redur May 21, 2024

stijnvermeeren-swisstopo May 21, 2024

redur May 21, 2024

stijnvermeeren-swisstopo left a comment

stijnvermeeren-swisstopo May 21, 2024

stijnvermeeren-swisstopo May 21, 2024

		from stratigraphy.util.line import TextLine


		class LayerIndexColumn:

		return blocks


		def matching_blocks(all_lines: list[TextLine], line_index: int, next_layer_index: TextLine \| None) -> list[TextBlock]:

Feat: Material Block recognition for Geneva Layout #45

Feat: Material Block recognition for Geneva Layout #45

Conversation

redur commented May 21, 2024

redur May 21, 2024 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented May 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stijnvermeeren-swisstopo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

redur May 21, 2024 •

edited

Loading

github-actions bot commented May 21, 2024 •

edited

Loading