Feature/most recent unique #2570

ivakegg · 2024-09-18T20:20:59Z

No description provided.

…e ivarator paths

warehouse/query-core/src/main/java/datawave/query/transformer/UniqueTransform.java

...ouse/query-core/src/main/java/datawave/query/util/sortedmap/BufferedFileBackedSortedMap.java

warehouse/query-core/src/main/java/datawave/query/transformer/DocumentTransform.java

warehouse/query-core/src/main/java/datawave/query/util/sortedmap/FileSerializableSortedMap.java

warehouse/query-core/src/main/java/datawave/query/util/sortedmap/HdfsBackedSortedMap.java

warehouse/query-core/src/main/java/datawave/query/util/sortedmap/FileKeyValueSortedMap.java

...y-core/src/main/java/datawave/query/util/sortedmap/rfile/KeyValueByteDocumentTransforms.java

warehouse/query-core/src/main/java/datawave/query/util/sortedmap/HdfsBackedSortedMap.java

warehouse/query-core/src/test/java/datawave/query/UniqueTest.java

FineAndDandy · 2025-01-09T14:41:24Z

warehouse/query-core/src/main/java/datawave/query/util/sortedmap/MultiMapBackedSortedMap.java

no unit tests for this class

hmm... perhaps not directly. I will see what I can do.

apmoriarty · 2025-01-09T14:42:42Z

warehouse/query-core/src/test/java/datawave/query/iterator/FieldIndexIntegrationTest.java

+            put("TLD_FIELD_A", 4); // 3 record ids, 1 value
+            put("TLD_FIELD_B", 14); // 13 record ids, 1 value
+            put("TLD_FIELD_C", 5); // 4 record ids, 1 value


What caused this assertion to change? Was the TLD aggregator not working before?

I really don't know why this changed.. I will figure it out however.

warehouse/query-core/src/main/java/datawave/query/util/sortedmap/MultiMapBackedSortedMap.java

FineAndDandy · 2025-01-09T19:18:03Z

warehouse/query-core/src/main/java/datawave/query/transformer/UniqueTransform.java

+        if (!this.uniqueFields.equals(uniqueFields)) {
+            this.uniqueFields = uniqueFields.clone();
+            log.info("Resetting unique fields on the unique transform");
+            this.bloom = BloomFilter.create(new ByteFunnel(), 500000, 1e-15);


since the bloom is only used when not using isMostRecent() can we avoid creating the bloom at all when that is set?

probably, I will see if I can do that.

...ouse/query-core/src/main/java/datawave/query/util/sortedmap/BufferedFileBackedSortedMap.java

ivakegg requested a review from jwomeara as a code owner September 18, 2024 20:20

Initial refactoring of the mostRecentUnique functionality to avoid th…

b98ad56

…e ivarator paths

ivakegg force-pushed the feature/mostRecentUnique branch from ca23bd7 to 33d30e5 Compare November 22, 2024 14:08

Initial implementation of a file backed sorted map

9dec45e

ivakegg force-pushed the feature/mostRecentUnique branch from da314e2 to 9dec45e Compare November 22, 2024 16:52

ivakegg added 2 commits November 22, 2024 20:28

Created additional document map for the return cache

9055168

Merge branch 'integration' into feature/mostRecentUnique

e8415da

lbschanno mentioned this pull request Dec 31, 2024

Add MAX_UNIQUE_COUNT functionality #2678

Open

ivakegg and others added 2 commits January 6, 2025 12:38

Merge branch 'integration' into feature/mostRecentUnique

fb6bcb1

GitHub Actions: Fix Formatting

04d46bf

ivakegg added the Integration Tested label Jan 8, 2025

avgAGB previously approved these changes Jan 8, 2025

View reviewed changes

FineAndDandy reviewed Jan 8, 2025

View reviewed changes

warehouse/query-core/src/main/java/datawave/query/transformer/UniqueTransform.java Show resolved Hide resolved

FineAndDandy reviewed Jan 8, 2025

View reviewed changes

...ouse/query-core/src/main/java/datawave/query/util/sortedmap/BufferedFileBackedSortedMap.java Show resolved Hide resolved