Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SpatialKNN model fails to initialize in a DLT, raising a Py4J Exception #431

Open
tigerhawkvok opened this issue Sep 12, 2023 · 0 comments

Comments

@tigerhawkvok
Copy link

Describe the bug

During a DLT's "setting up tables", flow resolution fails

image

Note the traceback points to

self.model = getattr(

To Reproduce
Steps to reproduce the behavior:

  1. Create a DLT pipeline
  2. In a table definition, use:
@dlt.table(
    partition_cols= ["countryCode"],
    table_properties= {"pipelines.autoOptimize.zOrderCols": "propertyID, distance"},
)
def _cellPeers() -> DataFramePS:
    # https://databrickslabs.github.io/mosaic/models/spatial-knn.html
    knn = SpatialKNN()
    if spark.sparkContext.getCheckpointDir():
        knn.setUseTableCheckpoint(True)
        knn.setCheckpointTablePrefix("checkpoint_table_knn")
   # anything else you want
   return foo

Expected behavior
No exception when initializing the class

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant