Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revert "Removed maximum size constraint when loading temp_table" #299

Merged
merged 1 commit into from
Dec 4, 2024
Merged

Revert "Removed maximum size constraint when loading temp_table" #299

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions digital_land/package/datasetparquet.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,9 +68,8 @@ def create_temp_table(self, input_paths):
# Initial max_line_size and increment step
max_size = 40000000
increment_step = 20000000
# max_limit = 200000000 # Maximum allowable line size to attempt
max_limit = 200000000 # Maximum allowable line size to attempt

increment = False
while True:
try:
query = f"""
Expand All @@ -85,13 +84,15 @@ def create_temp_table(self, input_paths):
)
"""
self.conn.execute(query)
if increment:
logging.info(f"Ended up needing a value of max_size = {max_size}")
break
except duckdb.Error as e: # Catch specific DuckDB error
if "Value with unterminated quote" in str(e):
increment = True
max_size += increment_step
if max_size > max_limit:
print(
f"Exceeded max_limit of {max_limit}. Could not resolve the issue."
)
raise
else:
raise

Expand Down
Loading