-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add streaming match generation #39
Conversation
else: | ||
isomorphism_candidates.append(result) | ||
return isomorphism_candidates | ||
def isomorphism_candidates(): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sorry for the quiet; doing a little benchmarking right now on the performance implications of calling a nested func vs unwrapping these!
[edit] (i.e. i totally agree that generators vs list comprehensions is a huge improvement if you abort early; but i'm less sure about if you still run to completion)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good. I tried to make the change as minimally invasive as possible but I agree that having them as separate functions would be cleaner.
Let me know if you come across anything that seems slower. In the worst case scenario, it goes through all the data and discovers there are no matches (i.e. a full scan).
I also think the grand project is a pretty cool idea, putting a networkx interface on different backends such as a SQL database. I could see where indexes could make full scan operations much faster.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@davidmezzetti I got about +10μs benchmarking locally with ASV, which I think is fair to say is in the noise :) nice work!
I'd LOVE to have better indexing in Grand; I've been aiming for feature-completeness first and then adding in "shortcuts" and optimizations; would love to chat about some gameplans if you're interested!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good to me. I think having a networkx dialect applied towards something like a SQLite backend with indexes could be a powerful in-memory graph solution.
Things like an attribute search could return immediately regardless if there was/wasn't data for example (i.e. no full scans). In the case of SQLite, you'd get the native code and smart memory management for free. But still have the ease-of-use of networkx.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
COMPLETELY agree, would love to have a sqlite true high speed implementation!
This PR proposes "streaming match generation" for
grandiso
.When working with large graphs and broad queries, the project currently has a number of places where results are queued up in lists. This requires having the full list at each step before proceeding to the next step. On top of that, if a
LIMIT
clause is specified, in some cases a large list needs to be built before the LIMIT clause is recognized and execution can be terminated.This PR is as minimally invasive to the current code as possible. In most cases, the iteration logic is replaced with an inner function that runs a generator. When these generators are chained together, it effectively walks through the graph iteratively and enables returning results in a "streaming" fashion.
There will be a corresponding PR in the GrandCypher project that will have a couple examples illustrating the full change.