Expand vertex file reader to parse particle type and momenta

[gipert]

This would make remage support a wide range of external event generators through RMGVertexFromFile. The implementation would need to be a bit more complex than what it is right now, since the input file structure is "jagged". I guess a lot of this is already implemented in RMGGrabmayrGCReader.

Actually, "vertex" in RMGVertexFromFile was originally intended to refer only to vertex position. We should maybe implement all of this as an additional generator.

[tdixon97]

I dont think the table necessarily needs to be jagged, we just need to include the evtid (and time?).

I think the option to read kinematics as a table like:

xpos,ypos,zpos, px,py,pz, particle, time, evtid (all as Arrays)

would be useful and only a relatively small modification of the current code. It might be nice to read LH5 not h5 files however.

[gipert]

Yes, I meant that the number of particles originating at the vertex is not fixed.

[tdixon97]

Started prototyping a vertex generator here: pyvertexgen.
It should be easy to write to a HDF5 file and start testing this.

[ManuelHu]

yes, but you will need to write it in geant4's ugly HDF5 ntuple format (are there even docs for it?). An LH5 reader would be very hard to add to the geant4 "analysis framework".

[tdixon97]

I don't know much about it, maybe we can write a simple converter like is done for the output?

[tdixon97]

I had a look and I am not sure how to create the Geant4 Ntuple inputs. Maybe we should go for something like the RMGGrabmayrGCReader as @gipert suggested and forget the geant4 input format.

[gipert]

• we want to use LH5
• no support for positions and momenta in the same table
• positions is simple flat table (x, y, z)
• momenta is still a flat table. Columns: evtid, g4_pid, px, py, pz

[tdixon97]

@ManuelHu what is the status of this? I recall you had something working

[ManuelHu]

not really. I have the general infrastructure to read LH5 files as input in a multithreaded run; but I did not yet work on the particle creation side. Only the vertex (position) reader works at the moment.

And I also still need to push some code I worked on locally, to really decouple the LH5 reading part from the vertex reader. After that, the rest should be rather easy.

[tdixon97]

I might be able to find some time to look into creating the vertices if the LH5 reader works. I think it should be fairly straightforward and I'd like to do this relatively soon.

[ManuelHu]

@tdixon: see #230 for the refactor. The RMGAnalysisReader can only be initialized from the main thread. Apart from that, it should be quite simple to use and adapt.

note: there is no sane way to access/check the structure of the input file with the G4*AnalysisReaders. The structure has always to be the fixed (columns might be optional though).

[tdixon97]

This looks great @ManuelHu , maybe you could add some docstrings to the main functions ? That would make it easier for people to use this reader for particular applications, and would be a step towards having proper documentation.
Can this read any LH5 file or just the geant4 ntuple format? We should definitely document what format the input file needs to be in.

[tdixon97]

I think we should also add some test on multithreading / thread safety since that makes me nervous..
We could generate some LH5 file of positions, read it in in multithread mode and then compare the output vertex table to the input ?

[ManuelHu]

Sure. I am also worried quite a lot, the whole situation is very bad:

• the analysis readers automatically mangle names when used on worker threads.
  • but we cannot pre-compute input files; as we don't know the number of rows needed
  • so we can only use them on the main thread
• analysis readers are thread-local by file type. there is no way to create a new reader that is not connected to this global instance
• So if we read two HDF5 files in different parts of remage that will be racy, unless we use a global lock around any analysisreader usage.
• ...and we just need to hope that Geant4 does not use it internally
• ouch

[tdixon97]

I guess we need to add some checks then?

…

________________________________ From: Manuel Huber ***@***.***> Sent: Tuesday, January 21, 2025 2:15 PM To: legend-exp/remage ***@***.***> Cc: Dixon, Toby ***@***.***>; Comment ***@***.***> Subject: Re: [legend-exp/remage] Expand vertex file reader to parse particle type and momenta (Issue #156) ⚠ Caution: External sender Sure. I am also worried quite a lot, the whole situation is very bad: * the analysis readers automatically mangle names when used on worker threads. * but we cannot pre-compute input files; as we don't know the number of rows needed * so we can only use them on the main thread * analysis readers are thread-local by file type. there is no way to create a new reader that is not connected to this global instance * So if we read two HDF5 files in different parts of remage that will be racy, unless we use a global lock around any analysisreader usage. * ...and we just need to hope that Geant4 does not use it internally * ouch — Reply to this email directly, view it on GitHub<#156 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ANSET44FEME4OMTCWQZK5KL2LZB77AVCNFSM6AAAAABRQCILQWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMBUG4YDINJZGM>. You are receiving this because you commented.Message ID: ***@***.***>

[ManuelHu]

I imagine an (ugly) solution. Maybe it works, but I would really test before that if we can actually concurrently read from two files before investing time into making this safe and simple to use.
If it does not work in that simple case (reading vtx and kin data), we would need to scrap the whole solution and implement the input reading by hand :-/