Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add streaming match generation #39

Merged
merged 1 commit into from
Feb 13, 2024
Merged

Add streaming match generation #39

merged 1 commit into from
Feb 13, 2024

Conversation

davidmezzetti
Copy link
Contributor

This PR proposes "streaming match generation" for grandiso.

When working with large graphs and broad queries, the project currently has a number of places where results are queued up in lists. This requires having the full list at each step before proceeding to the next step. On top of that, if a LIMIT clause is specified, in some cases a large list needs to be built before the LIMIT clause is recognized and execution can be terminated.

This PR is as minimally invasive to the current code as possible. In most cases, the iteration logic is replaced with an inner function that runs a generator. When these generators are chained together, it effectively walks through the graph iteratively and enables returning results in a "streaming" fashion.

There will be a corresponding PR in the GrandCypher project that will have a couple examples illustrating the full change.

else:
isomorphism_candidates.append(result)
return isomorphism_candidates
def isomorphism_candidates():
Copy link
Member

@j6k4m8 j6k4m8 Feb 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry for the quiet; doing a little benchmarking right now on the performance implications of calling a nested func vs unwrapping these!

[edit] (i.e. i totally agree that generators vs list comprehensions is a huge improvement if you abort early; but i'm less sure about if you still run to completion)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. I tried to make the change as minimally invasive as possible but I agree that having them as separate functions would be cleaner.

Let me know if you come across anything that seems slower. In the worst case scenario, it goes through all the data and discovers there are no matches (i.e. a full scan).

I also think the grand project is a pretty cool idea, putting a networkx interface on different backends such as a SQL database. I could see where indexes could make full scan operations much faster.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davidmezzetti I got about +10μs benchmarking locally with ASV, which I think is fair to say is in the noise :) nice work!

I'd LOVE to have better indexing in Grand; I've been aiming for feature-completeness first and then adding in "shortcuts" and optimizations; would love to chat about some gameplans if you're interested!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good to me. I think having a networkx dialect applied towards something like a SQLite backend with indexes could be a powerful in-memory graph solution.

Things like an attribute search could return immediately regardless if there was/wasn't data for example (i.e. no full scans). In the case of SQLite, you'd get the native code and smart memory management for free. But still have the ease-of-use of networkx.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

COMPLETELY agree, would love to have a sqlite true high speed implementation!

@j6k4m8 j6k4m8 merged commit 7fc2b32 into aplbrain:master Feb 13, 2024
5 checks passed
@davidmezzetti davidmezzetti deleted the match-streaming branch February 13, 2024 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants