L1TMuonEndCapTrackProducer::produce() takes 96 MB memory per stream

[makortel]

Live memory profiles of #40437 (comment) show that L1TMuonEndCapTrackProducer::produce() takes 100 MB memory / stream

• 1 thread and stream https://mkortela.web.cern.ch/mkortela/cgi-bin/navigator/issue40437/reco_07.10_live/240
• 4 threads and streams https://mkortela.web.cern.ch/mkortela/cgi-bin/navigator/issue40437/reco_4th_4st_07.10_live/101

The L1TMuonEndCapTrackProducer module itself is an edm::stream. The memory consumption can be split into

• 93 MB / stream in PtAssignmentEngine::load(), that seems to load some kind of BDT?
• 1.8 MB / stream in EMTFSetup::reload() loading some LUTs
• 0.5 MB / stream in SectorProcessor::process() that is mostly Tensorflow

Assuming the PtAssignmentEngine::load() is indeed BDT or similar, does its representation really need to be that large? Ideally all of these would be in GlobalCache of the module (e.g. Tensorflow stuff), or in the EventSetup (e.g. the BDT whose content apparently depends on the L1TMuonEndCapParams and L1TMuonEndCapForest EventSetup data products).

[makortel]

assign l1

[cmsbuild]

New categories assigned: l1

@epalencia,@aloeliger you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

Moving the Tensorflow stuff to GlobalCache was also discussed in #32894

[eyigitba]

Hi @makortel , as you said the main contribution here is loading of the BDT in PtAssignmentEngine::load and coordinate conversion LUTs in SectorProcessorLUT::read. I don't know how we can reduce these easily.

Regarding the Tensorflow stuff and GlobalCache, I currently don't have the time to rework the code unfortunately. If you and/or L1T offline software think that this should be done, I can try to pass this task to someone within the EMTF group and we can see how quickly we can implement this.

[makortel]

Hi @eyigitba I think addressing the PtAssignmentEngine::load() (at least making read-only parts shared across streams; I hope the memory there is mostly read-only) would be important.

The Tensorflow part would be nice (e.g. if the model would become larger in the future, or serving as an example for others), but with 0.5 MB / stream not that important today.

[eyigitba]

Hi @makortel , ok we'll look into the BDT loading and also see how we can improve the Tensorflow part.

How urgent is this btw?

[VinInn]

Memory budget is 2GB per stream (including shared component and I/O buffers).
L1TMuonEndCapTrackProducer alone accounts for 5% of that.
To L1 mngmt to judge how urgent it WAS.

[VinInn]

BTW: why using your own implementation of a Forest and not the highly optimized common CMS one
https://cmssdt.cern.ch/dxr/CMSSW/source/CondFormats/GBRForest/interface/GBRForest.h#24
?

I would advice to switch to that.

[eyigitba]

Thanks for the advice @VinInn . I don't know why it was implemented like this, but this code is quite old, from 2016 or so. I unfortunately don't have much time these couple of weeks, but we'll discuss in the EMTF group to come up with a solution soon.

[makortel]

The PtAssignmentEngine::load() came up again in #46040 (comment), this time with ~47 MB / stream cost. On an 8-thread/stream PromptReco that corresponds to 376 MB. Avoiding the per-stream copies could potentially save ~329 MB, and moving to GBRForest would likely save more.

In my opinion it would be great if this would be addressed for 2025 data taking.

[cmsbuild]

cms-bot internal usage

[makortel]

type performance-improvements

[makortel]

@eyigitba @cms-sw/l1-l2 Any news?

[eyigitba]

@makortel , not yet from my side. I have a student looking into this but nothing to discuss for now

[makortel]

Came up in #46975 (comment)