feat: support get_option for local-rel threshold (#65)

SparkConnect's Scala client currently fails for spark.createDataFrame() requests with a `NotImplementedError`. The Scala client for this API submits a request to `"get_option"` with key `"localRelationCacheThreshold"`. We only support ConfigRequests for this key if `"op_type"` is `"get"`. fixes #61 Co-authored-by: pat70 <[email protected]>
voltrondata · Aug 3, 2024 · a6e0673 · a6e0673
1 parent 07d3661
commit a6e0673
Showing 1 changed file with 3 additions and 0 deletions.
diff --git a/src/gateway/server.py b/src/gateway/server.py
@@ -354,6 +354,9 @@ def Config(self, request, context):
                 for key in request.operation.get_option.keys:
                     if key == 'spark.sql.session.timeZone':
                         response.pairs.add(key=key, value='UTC')
+                    elif key in ['spark.sql.session.localRelationCacheThreshold',
+                                 'spark.sql.repl.eagerEval.maxNumRows']:
+                        response.pairs.add(key=key, value='9999')
                     else:
                         _LOGGER.info(f'Unknown config item: {key}')
                         raise NotImplementedError(f'Unknown config item: {key}')