Implement query validation endpoint #2645

lbschanno · 2024-11-21T06:45:24Z

Implement the query/{logicName}/validate endpoint. This feature supports the ability to configure validation rules that will validate LUCENE and JEXL queries against a number of criteria and provide meaningful feedback to customers.

Closes #2585

Each validation criteria is implemented via a corresponding datawave.query.rules.QueryRule implementation.

Rule implementations:

`datawave.query.rules.AmbiguousNotRule`

Checks LUCENE queries for usage of NOT without using parens (NOT takes precedence so the resulting plan could be quite incorrect):

FIELD1:abc OR FIELD2:def NOT FIELD3:123 -> should be (FIELD1:abc OR FIELD2:def) NOT FIELD3:123

Sample bean config:

<bean id="AmbiguousNotRule" scope="prototype" class="datawave.query.rules.AmbiguousNotRule">  
    <property name="name" value="Check for Ambiguous Usage of NOT"/>  
</bean>

Sample message for query FIELD1:abc FIELD2:def NOT FIELD:ghi:

Ambiguous usage of NOT detected with multiple unwrapped preceding terms: "FIELD1:abc AND FIELD2:def NOT" should be "(FIELD1:abc AND FIELD2:def) NOT".

`datawave.query.rules.AmbiguousOrPhrasesRule`

Checks LUCENE queries for any parens before field or completely missing parens when using ORs:

FIELD:1234 OR 5678 -> should be FIELD:(1234 OR 5678)
(FIELD:1234 OR 5678) -> should be FIELD:(1243 OR 5687)

Sample bean config:

<bean id="AmbiguousOrPhrasesRule" scope="prototype" class="datawave.query.rules.AmbiguousOrPhrasesRule">  
    <property name="name" value="Check for Unfielded Terms That Could Be Wrapped"/>  
</bean>

Sample message for query FOO:abc OR def OR efg:

Ambiguous unfielded terms OR'd with fielded term detected: FOO:abc OR def OR efg Recommended: FOO:(abc OR def OR efg)

`datawave.query.rules.AmbiguousUnquotedPhrasesRule`

Checks LUCENE queries for any phrases not quoted (where we would automatically AND) anywhere that the boolean operator is missing

FIELD:term1 term2 -> should be FIELD:"term1 term2"

Sample bean config:

<bean id="AmbiguousUnquotedPhrasesRule" scope="prototype" class="datawave.query.rules.AmbiguousUnquotedPhrasesRule">  
    <property name="name" value="Check for Unfielded Terms That Could Be Quoted"/>  
</bean>

Sample message for query FOO:abc def ghi:

Ambiguous unfielded terms AND'd with fielded term detected: FOO:abc AND def AND ghi. Recommended: FOO:"abc def ghi"

`datawave.query.rules.FieldExistenceRule`

Checks the query for any fields that do not exist in the data dictionary. Special fields may be specified that will not be flagged if they do not exist in the data dictionary.
Sample bean config:

<bean id="FieldExistenceRule" scope="prototype" class="datawave.query.rules.FieldExistenceRule">  
    <property name="name" value="Check Field Existence"/>  
    <property name="specialFields">  
        <list value-type="java.lang.String">   
            <value>_ANYFIELD_</value>  
            <value>_NOFIELD_</value>  
        </list>  
    </property>  
</bean>

Sample message for query FOO:abc AND MISSING1:def AND MISSING2:efg:

Fields not found in data dictionary: MISSING1, MISSING2

`datawave.query.rules.FieldPatternPresenceRule`

Returns configured messages if we see field X or Y.
Sample bean config:

<bean id="FieldPatternPresenceRule" scope="prototype" class="datawave.query.rules.FieldPatternPresenceRule">  
    <property name="name" value="Check Presence of Field or Pattern"/>  
    <property name="fieldMessages">  
        <map key-type="java.lang.String" value-type="java.lang.String">  
            <entry key="_ANYFIELD_" value="Detected presence of _ANYFIELD_"/>  
        </map>  
    </property>  
    <property name="patternMessages">  
        <map key-type="java.lang.String" value-type="java.lang.String">  
            <entry key=".*" value="Detected pattern '.*' that will match everything"/>  
        </map>  
    </property>  
</bean>

Sample messages for query _ANYFIELD_:abc || FOO:.*

Detected presence of _ANYFIELD_
Detected pattern '.*' that will match everything

`datawave.query.rules.IncludeExcludeArgsRule`

Checks LUCENE queries for any #INCLUDE or #EXCLUDE functions with an incorrect number of arguments, taking into account whether a boolean was given as the first argument.
Sample bean config:

<bean id="IncludeExcludeArgsRule" scope="prototype" class="datawave.query.rules.IncludeExcludeArgsRule">  
    <property name="name" value="Validate Args of #INCLUDE and #EXCLUDE"/>  
</bean>

Sample message for query: #INCLUDE(OR, 'value')

Function #INCLUDE supplied with uneven number of arguments after the first boolean arg OR. Must supply field/value after the boolean, e.g. #INCLUDE(OR, FIELD, 'value') or #INCLUDE(OR, FIELD1, 'value1',' FIELD2, 'value2').

`datawave.query.rules.IncludeExcludeIndexFieldsRule`

Checks queries for any usage of #INCLUDE or #EXCLUDE with index fields in them.
Sample bean config:

<bean id="IncludeExcludeIndexFieldsRule" scope="prototype" class="datawave.query.rules.IncludeExcludeIndexFieldsRule">  
    <property name="name" value="Check #INCLUDE and #EXCLUDE for Indexed Fields"/>  
</bean>

Sample message for query: FOO:abc AND #INCLUDE(INDEXED1, 'abc'):

Indexed fields found within the function filter:includeRegex: INDEXED1

`datawave.query.rules.InvalidQuoteRule`

Checks LUCENE queries for the usage of the quote ` instead of '.
Sample bean config:

<bean id="InvalidQuoteRule" scope="prototype" class="datawave.query.rules.InvalidQuoteRule">  
    <property name="name" value="Check for Invalid Quote"/>  
</bean>

Sample message for query FOO:'abc' AND BAR:`def` :

Invalid quote ` used in phrase "BAR:`abc`". Use ' instead.

`datawave.query.rules.MinimumSlopProximityRule`

Checks queries for any cases where the proximity is smaller than the number of terms.

FIELD:"term1 term2 term3"~1 -> the 1 should be 3 or greater.

Sample bean config:

<bean id="MinimumSlopProximityRule" scope="prototype" class="datawave.query.rules.MinimumSlopProximityRule">  
    <property name="name" value="Validate Slop Proximity"/>  
</bean>

Sample message for query FOO:"term1 term2 term3"~2:

Invalid slop proximity, the 2 should be 3 or greater: FOO:"term1 term2 term3"~2

`datawave.query.rules.NumericValueRule`

Checks queries for any cases where a numeric value is used against a non-numeric field.
Sample bean config:

<bean id="NumericValueRule" scope="prototype" class="datawave.query.rules.NumericValueRule">  
    <property name="name" value="Validate Numeric Values Only Given for Numeric Fields"/>  
</bean>

Sample message for query AGE:1 AND NON_NUMERIC_FIELD:4

Numeric values supplied for non-numeric field(s): NON_NUMERIC_FIELD

`datawave.query.rules.TimeFunctionRule`

Checks queries for any usage of #TIME_FUNCTION with any non-date fields.
Sample bean config:

<bean id="TimeFunctionRule" scope="prototype" class="datawave.query.rules.TimeFunctionRule">  
    <property name="name" value="Validate #TIME_FUNCTION has Date Fields"/>  
</bean>

Sample message for query filter:timeFunction(DATE1,NON_DATE2,'-','>',2522880000000L):

Function #TIME_FUNCTION (filter:timeFunction) found with fields that are not date types: NON_DATE2

`datawave.query.rules.UnescapedSpecialCharsRule`

Checks queries for fields and patterns with unescaped special characters. Characters that should be considered exceptions to the rule may be configured for literals and patterns. The regex-reserved characters . + * ? ^ $ ( ) [ ] { } | \ will always be treated as exceptions for patterns. Whether whitespace characters should be considered special characters is also configurable.
Sample bean config:

<bean id="UnescapedSpecialCharsRule" scope="prototype" class="datawave.query.rules.UnescapedSpecialCharsRule">  
    <property name="name" value="Check for Unescaped Special Characters"/>  
    <property name="literalExceptions">  
        <list value-type="java.lang.Character">  
            <value>?</value>  
        </list>  
    </property>  
    <property name="patternExceptions">  
        <list value-type="java.lang.Character">  
            <value>_</value>  
        </list>  
    </property>  
    <property name="escapedWhitespaceRequiredForLiterals" value="false"/>  
    <property name="escapedWhitespaceRequiredForPatterns" value="false"/>  
</bean>

Sample message for query FOO == 'ab?c' || FOO =~ 'ab_cd':

Literal string "ab?c" has the following unescaped special character(s): '?'");  
Regex pattern "ab_cd" has the following unescaped special character(s): '_'"

`datawave.query.rules.UnescapedWildcardsInPhrasesRule`

Checks LUCENE queries for any quoted phrases that have an unescaped wildcard in them.
Sample bean config:

<bean id="UnescapedWildcardsInPhrasesRule" scope="prototype" class="datawave.query.rules.UnescapedWildcardsInPhrasesRule">  
    <property name="name" value="Check Quoted Phrases for Unescaped Wildcard"/>  
</bean>

Sample message for FOO:"ab*":

Unescaped wildcard found in phrase FOO:"ab*". Wildcard is incorrect, or phrase should be FOO:/ab*/

`datawave.query.rules.UnfieldedTermsRule`

Checks LUCENE queries for any unfielded terms.
Sample bean config:

<bean id="UnfieldedTermsRule" scope="prototype" class="datawave.query.rules.UnfieldedTermsRule">  
    <property name="name" value="Check for Unfielded Terms"/>  
</bean>

Sample message for FOO:abc def "efg"

Unfielded term def found.
Unfielded term "efg" found.

Activating Rules

Rules must be configured in the QueryLogicFactory.xml and added to the list of rules configured for the validationRules property of a ShardQueryLogic.
Sample bean config:

	<!-- Rules added to validationRules via referencing bean definition samples. -->
    <property name="docAggregationThresholdMs" value="-1" />  
    <property name="tfAggregationThresholdMs" value="-1" />  
    <property name="validationRules">  
        <list value-type="datawave.query.rules.QueryRule">  
            <ref bean="InvalidQuoteRule"/>  
            <ref bean="UnfieldedTermsRule"/>  
            <ref bean="UnescapedWildcardsInPhrasesRule"/>  
            <ref bean="AmbiguousNotRule"/>  
            <ref bean="AmbiguousOrPhrasesRule"/>  
            <ref bean="AmbiguousUnquotedPhrasesRule"/>  
            <ref bean="MinimumSlopProximityRule"/>  
            <ref bean="IncludeExcludeArgsRule"/>  
            <ref bean="FieldExistenceRule"/>  
            <ref bean="UnescapedSpecialCharsRule"/>  
            <ref bean="FieldPatternPresenceRule"/>  
            <ref bean="IncludeExcludeIndexFieldsRule"/>  
            <ref bean="NumericValueRule"/>  
            <ref bean="TimeFunctionRule"/>  
        </list>  
    </property>  
</bean>

Some notes on rules:

All rules may be configured with a name. Names are not required to be unique. If a name is not specified, the name will be the name of the rule class.
Rules will be executed in the order that they are listed in the validationRules property.
Rules will never be executed twice for a validation operation.
Some rules will only execute against LUCENE queries. The rest will execute on the resulting parsed JEXL query. LUCENE-only rules:

AmbiguousNotRule
AmbiguousOrRule
AmbiguousUnquotedPhrasesRule
IncludeExcludeArgsRule
InvalidQuoteRule
MinimumSlopProximityRule
UnescapedWildcardsInPhrasesRule
UnfieldedTermsRule

REST Response Schema

The REST response schema will be as follows:

{
	// A list of names of rules that successfully executed 
	// against the query, in the order that they executed.
	"ExecutedRules": [
		"<RuleName>",
		// ...
	],
	// Whether any issues were flagged.
	"HasResults": true,
	"LogicName": "<logicName>",
	"OperationTimeMS": 10,
	"QueryId": "<uuid>",
	// The flagged issues in the order that they were flagged.
	"Results": [
		{
			"RuleName":"<RuleName>",
			"Messages":[
				"A message about a flagged issue",
				// ...
			],
			// This is populated only if an exception occurred 
			// while a rule was attempting to validate a query.
			"Exception": {
				"Cause": "",
				"Message":""
			}
		},
		// ...
	],
	"Exceptions": [
		{
			"Cause":"",
			"Code":"",
			"Message":""
		},
		// ...
	]
}

Implement the query/{logicName}/validate endpoint. This feature supports the ability to configure validation rules that will validate LUCENE and JEXL queries against a number of criteria and provide meaningful feedback to customers. Closes #2585

warehouse/query-core/src/main/java/datawave/query/language/parser/QueryParser.java

.../query-core/src/main/java/datawave/query/language/parser/jexl/JexlControlledQueryParser.java

core/query/src/main/java/datawave/core/query/logic/QueryLogic.java

warehouse/query-core/src/main/java/datawave/query/tables/ShardQueryLogic.java

web-services/query/src/main/java/datawave/webservice/query/runner/QueryExecutorBean.java

warehouse/query-core/src/main/java/datawave/query/tables/ShardQueryLogic.java

...se/query-core/src/main/java/datawave/query/jexl/visitors/FieldsWithNumericValuesVisitor.java

.../query-core/src/main/java/datawave/query/lucene/visitors/AmbiguousUnfieldedTermsVisitor.java

warehouse/query-core/src/main/java/datawave/query/lucene/visitors/BaseVisitor.java

warehouse/query-core/src/main/java/datawave/query/lucene/visitors/PrintingVisitor.java

ivakegg

A few comments to handle, but all in all this looks pretty good! Created #2656 with some of the suggested changes.

* Suggested changes * Removed unused imports

Implement query validation endpoint

08a8df4

Implement the query/{logicName}/validate endpoint. This feature supports the ability to configure validation rules that will validate LUCENE and JEXL queries against a number of criteria and provide meaningful feedback to customers. Closes #2585

lbschanno requested review from apmoriarty, ivakegg and jwomeara November 21, 2024 06:45

lbschanno mentioned this pull request Nov 21, 2024

query validation endpoint #2585

Open

lbschanno and others added 4 commits November 21, 2024 04:06

Fix failing unit test

bb0bc48

Merge branch 'integration' into task/queryValidationEndpointImpl

95d1558

Fix javadoc

a2d24a4

Make InvalidQuoteVisitor.INVALID_QUOTE a primitive

c64b9d2