[DAQ] Input source improvements and initial Phase-2 DTH format support #47068

smorovic · 2025-01-09T15:14:26Z

PR description:

Adds several major features for FedRawDataSource and/or DAQSource:

deletion of files moves to fileDeleter thread with new throttling check to limit number of files that are not yet deleted. Reworked also (complementary) end-of-LS detection using tbb concurrent hash map and tracking lumisections having files being processed.
DAQSource: DTH unpacking (raw format, currently not containing FRD-style headers). Inncluding dummy data generator and an unit test
DTH Phase2 format definitions (preliminary). The format is not yet final and at the moment definition will remain in EventFilter/Utilities. When finalized and tested with real payload, it will be be moved to DataFormats.
DAQSource: FRDPreUnpack mode which moves event unpacking into I/O threads and reducing time spent in read event which is serialized for all CMSSW streams.
dropping support for very old (pre-Run3) FRD format versions (any such files can be read and converted with an older CMSSW version). Unused single-buffer source support was also dropped.
improved directory handling and run number detection in file list mode
improved synchronization (less sleeping) between data fetching and main (framework) thread in sources

PR validation:

new unit tests added
tested in a small scale DAQ F3 test setup (3 BU/FU VMs)
tested in CDAQ emulator runs (several hours)

* adding the missing JSON field to test RAW writer * termination condition can skip writing files (with service used to detect it) * change ls type to unsigned * remove old commented section fixes to fileListMode apply equivalent changes to DAQSource strip singleBufferMode

improved synchronization between threads (less sleeping)

fix variable type

…point

…e reader thread. API changes and simplifications are propagated to all models. bugfixes to new mode removing debug log updated unit test

- preliminary header definition of DTH Orbit, DTH FragmentTrailer, SLinExpress fragment header and trailer. - new module for DAQSource to read raw DTH dumps. They must be grouped (sorted)together by orbit number, contain event fragments in the same sequence and one orbit must fit into the buffer. - new DTH dummy raw generator - FRD dummy raw generator updated, dropping generation of obsolete FRD versions - this is compatible with orbit-building in EVB, a FRD-like header could be added in the future It will be followed up by models allowing separate striped multi-orbit files (similar to SFB/FRDStriped or L1Scouting-like approach)

…RD versions < 5

…pre-unpack

cmsbuild · 2025-01-09T15:14:57Z

cms-bot internal usage

cmsbuild · 2025-01-09T15:16:22Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47068/43236

ERROR: Build errors found during clang-tidy run.

src/EventFilter/Utilities/src/DAQSourceModelsDTH.cc:132:22: error: equality comparison result unused [clang-diagnostic-unused-comparison]
  132 |       checksumError_ == std::string();
      |       ~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~
src/EventFilter/Utilities/src/DAQSourceModelsDTH.cc:132:22: note: use '=' to turn this equality comparison into an assignment
--
gmake: *** [config/SCRAM/GMake/Makefile.coderules:129: code-checks] Error 2
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 2

…ased tracking which isn't fully guaranteed to close file in timely manner (e.g. a stream that last processed an older file could not get event for a long time potentially and we'd keep that file open for extended time).

cmsbuild · 2025-01-09T15:21:05Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47068/43237

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-checks:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47068/43237/code-checks.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47068/43237/code-checks.patch | patch -p1
You can also run scram build code-checks to apply code checks directly
code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47068/43237/code-format.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47068/43237/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2025-01-09T22:08:47Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47068/43247

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47068/43247/code-format.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47068/43247/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2025-01-09T22:14:42Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47068/43249

cmsbuild · 2025-01-09T22:15:05Z

Pull request #47068 was updated.

smorovic · 2025-01-09T22:23:17Z

@cmsbuild please test

cmsbuild · 2025-01-10T00:54:17Z

+1

Size: This PR adds an extra 476KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9c7702/43709/summary.html
COMMIT: 06fdce0
CMSSW: CMSSW_15_0_X_2025-01-09-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/47068/43709/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9c7702/43709/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9c7702/43709/git-merge-result

Comparison Summary

Summary:

You potentially removed 131 lines from the logs
Reco comparison results: 1 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3818730
DQMHistoTests: Total failures: 45
DQMHistoTests: Total nulls: 13
DQMHistoTests: Total successes: 3818652
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.8560000000000004 KiB( 48 files compared)
DQMHistoSizes: changed ( 1000.0,... ): -0.012 KiB MessageLogger/Errors
DQMHistoSizes: changed ( 1000.0,... ): -0.012 KiB MessageLogger/Warnings
DQMHistoSizes: changed ( 24834.911,... ): -0.008 KiB MessageLogger/Errors
DQMHistoSizes: changed ( 24834.911,... ): -0.008 KiB MessageLogger/Warnings
DQMHistoSizes: changed ( 4.22,... ): -0.004 KiB MessageLogger/Errors
DQMHistoSizes: changed ( 4.22,... ): -0.004 KiB MessageLogger/Warnings
Checked 214 log files, 184 edm output root files, 49 DQM output files
TriggerResults: no differences found

cmsbuild · 2025-01-12T00:35:34Z

A new Pull Request was created by @smorovic for master.

It involves the following packages:

EventFilter/Utilities (daq)

@emeschi, @smorovic can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @missirol this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

smorovic · 2025-01-12T00:41:13Z

+daq

cmsbuild · 2025-01-12T00:41:40Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @mandrenguyen, @rappoccio, @sextonkennedy, @antoniovilela (and backports should be raised in the release meeting by the corresponding L2)

mandrenguyen · 2025-01-12T08:51:13Z

+1

smorovic added 11 commits January 9, 2025 11:38

file deleter thread instead of DAQ director

7a99b4b

improved synchronization between threads (less sleeping)

port deleter thread and thread synchronization changes to DAQSource

2f81cb5

fix variable type

fix two deadlock scenarios

e1ecb24

fixes to DAQSource striped mode and EvFDaqDirector detection of mount…

667935c

…point

add new fields to EoR json of the (unit) test writer

e727c63

New DAQSourceModel "PreUnpack" where FRD collection is unpacked in th…

ebcb616

…e reader thread. API changes and simplifications are propagated to all models. bugfixes to new mode removing debug log updated unit test

update error conditions and monitoring for new file deletion thread

0693cc1

*port held files monitoring to DAQSource, drop support for very old F…

9ae4ec3

…RD versions < 5

fix FRDPreUnpack to not modify file structure variables when running …

d976961

…pre-unpack

cmsbuild added this to the CMSSW_15_0_X milestone Jan 9, 2025

cmsbuild added daq-pending pending-signatures tests-pending orp-pending code-checks-pending labels Jan 9, 2025

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jan 9, 2025

smorovic added 4 commits January 9, 2025 16:17

add README for DTH source

59f07e1

fix header ifdef

70a3af4

fix leftover asserts and clean up commented code

7ed2304

smorovic force-pushed the 15_0_X-source-improvements-2 branch from d63822b to 7ed2304 Compare January 9, 2025 15:19

cmsbuild added code-checks-pending and removed code-checks-rejected labels Jan 9, 2025

cmsbuild removed the code-checks-pending label Jan 9, 2025

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jan 9, 2025

reorganize input file and chunk classes in separate header

06fdce0

smorovic force-pushed the 15_0_X-source-improvements-2 branch from 894d6b3 to 06fdce0 Compare January 9, 2025 22:12

cmsbuild added code-checks-pending and removed code-checks-rejected labels Jan 9, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Jan 9, 2025

cmsbuild added tests-started and removed tests-pending labels Jan 9, 2025

cmsbuild added tests-approved and removed tests-started labels Jan 10, 2025

smorovic marked this pull request as ready for review January 12, 2025 00:35

cmsbuild added daq-approved fully-signed and removed daq-pending pending-signatures labels Jan 12, 2025

cmsbuild added orp-approved and removed orp-pending labels Jan 12, 2025

cmsbuild merged commit a44b537 into cms-sw:master Jan 12, 2025
11 checks passed

cmsbuild mentioned this pull request Jan 12, 2025

update hls4mlEmulatorExtras to v1.1.4 cms-sw/cmsdist#9606

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DAQ] Input source improvements and initial Phase-2 DTH format support #47068

[DAQ] Input source improvements and initial Phase-2 DTH format support #47068

smorovic commented Jan 9, 2025 •

edited

Loading

cmsbuild commented Jan 9, 2025 •

edited

Loading

cmsbuild commented Jan 9, 2025

cmsbuild commented Jan 9, 2025

cmsbuild commented Jan 9, 2025

cmsbuild commented Jan 9, 2025

cmsbuild commented Jan 9, 2025

smorovic commented Jan 9, 2025

cmsbuild commented Jan 10, 2025

cmsbuild commented Jan 12, 2025

smorovic commented Jan 12, 2025

cmsbuild commented Jan 12, 2025

mandrenguyen commented Jan 12, 2025

[DAQ] Input source improvements and initial Phase-2 DTH format support #47068

[DAQ] Input source improvements and initial Phase-2 DTH format support #47068

Conversation

smorovic commented Jan 9, 2025 • edited Loading

PR description:

PR validation:

cmsbuild commented Jan 9, 2025 • edited Loading

cmsbuild commented Jan 9, 2025

cmsbuild commented Jan 9, 2025

cmsbuild commented Jan 9, 2025

cmsbuild commented Jan 9, 2025

cmsbuild commented Jan 9, 2025

smorovic commented Jan 9, 2025

cmsbuild commented Jan 10, 2025

Comparison Summary

cmsbuild commented Jan 12, 2025

smorovic commented Jan 12, 2025

cmsbuild commented Jan 12, 2025

mandrenguyen commented Jan 12, 2025

smorovic commented Jan 9, 2025 •

edited

Loading

cmsbuild commented Jan 9, 2025 •

edited

Loading