command line tool support for vector search #175
Conversation
Thejas-bhat
commented
Nov 3, 2023
•
Thejas-bhat
commented
- also includes fixing the docvalues parsing and fields tool parsing given the new index sections format.
Thejas-bhat
force-pushed
the
cmdTooling
branch
from
ef7e51a to
f64db35
Compare
Thejas-bhat
requested review from
abhinavdangeti,
metonymic-smokey,
CascadingRadium,
Likith101 and
moshaad7
There was a problem hiding this comment.
@Thejas-bhat it'd be ideal if you could re-use methods rather than re-implement them again for the command line tooling. It'd be double work for us to maintain this otherwise. Is there any code here that's common with access methods elsewhere?
There was a problem hiding this comment.
@Thejas-bhat any thing else that needs look into in light of my earlier comment?
cmd/zap/cmd/vector.go
Outdated
| faiss.MetricLp: "lp distance", | ||
| faiss.MetricCanberra: "canberra distance", | ||
| faiss.MetricBrayCurtis: "bray curtis distance", | ||
| faiss.MetricJensenShannon: "jensen shannon distance", |
There was a problem hiding this comment.
We don't support any of these other than first 2, so no need to accommodate these.
There was a problem hiding this comment.
This has been marked as resolved but not actually resolved.
There was a problem hiding this comment.
sorry about that, i think i forgot to save the files before the push :p
cmd/zap/cmd/root.go
Outdated
| const ( | ||
| sectionInvertedTextIndex = iota | ||
| sectionFaissVectorIndex | ||
| ) |
There was a problem hiding this comment.
We don't need to redeclare these here, use zap.sectionInvertedTextIndex and zap.sectionFaissVectorIndex
There was a problem hiding this comment.
so i'd have to export those two vars in section.go. i've made the change, please let me know that's fine.
Thejas-bhat
force-pushed
the
cmdTooling
branch
from
f7c5e7d to
f4a96c2
Compare
i've reused a bunch of functions, however the diff shows a lot of change mainly because i moved the content of the docvalue command to a separate function. |
cmd/zap/cmd/explore.go
Outdated
| @@ -28,6 +28,7 @@ import ( | |||
| const termNotEncoded = 0 | |||
|
|
|||
| // exploreCmd represents the explore command | |||
| // no big changes over here as well. | |||
cmd/zap/cmd/vector.go
Outdated
| faiss.MetricLp: "lp distance", | ||
| faiss.MetricCanberra: "canberra distance", | ||
| faiss.MetricBrayCurtis: "bray curtis distance", | ||
| faiss.MetricJensenShannon: "jensen shannon distance", |
There was a problem hiding this comment.
This has been marked as resolved but not actually resolved.
Thejas-bhat
force-pushed
the
cmdTooling
branch
from
f4a96c2 to
e21aba6
Compare
Thejas-bhat
force-pushed
the
cmdTooling
branch
from
e21aba6 to
2688a22
Compare
* command line support for vector section - initial commit * docvalue tool fixes * bug fix: docvalue cmd returning nil result * refactoring fields cmd tool * code cleanup
