Extend database to a group of seeded runs as largest entity

[ambros-gleixner]

No description provided.

[fschloesser]

The plan is to create a group model that contains individual testruns, that the user can freely select.
Now a some fundamental questions arise:

1. Can a group contain other groups, meaning a testrun is treated as a group with one element? (Note: This does not pose a problem to seeded permutation runs.)
2. Do single testruns belong to one group only or can they be part of more than one group?
3. How to display a simple overview without redundancies (as one may want to compare groups with single testruns)? (Next to groups, show all testruns or only testruns that are in no group?)
4. How to display and order groups, what information does a group contain?

[ambros-gleixner]

1. Can a group contain other groups,

I don't know what multi-level groups would mean or be used for, except as the union of all testruns. (Do you? I thought for a second that a group of groups could mean a comparison of groups, but I think we should not use groups as a a way to store an evaluation.)

meaning a testrun is treated as a group with one element? (Note: This does not pose a problem to seeded permutation runs.)

Yes, we could consider testruns as singleton groups. And maybe related:

1a. Should a merge/union operation for groups be possible? I.e., should we be able to construct a new group as union of existing groups. Yes. Should we be able to easily add all testruns from another group? Yes.

2. Do single testruns belong to one group only or can they be part of more than one group?

It would be useful if a testrun can be added to more than one group. In this case, maybe groups should not be viewed as a top-level entity above test runs, but rather "orthogonal"?

We should avoid creating copies of testruns in order to include them in several groups. So a group should maybe more be a list of links to testruns.

Important consideration: deletion. Testruns should be deleted if and only if they are not contained in any group anymore. There should be a warning if that happens.

4. How to display and order groups, what information does a group contain?

In my mind, a group could be nothing more than a list of (links to) test runs. I don't know any information it could store that is not well-defined by the testruns. Of course, it could make sense for the implementation, if they store some info on the testruns for speed or convenience, but that is too technical for me to judge.

3. How to display a simple overview without redundancies (as one may want to compare groups with single testruns)? (Next to groups, show all testruns or only testruns that are in no group?)

One idea: have the same table view like now, each row being a group. In the columns we display

• the value if all testruns have the same value
• the list of values from all testruns if there are multiples; or maybe a * for some columns if width would get excessive

Ideally: one click would dynamically extend the row to display one row for each testrun contained with data like now.

[ambros-gleixner]

I realized another possible requirement that I want to mention before I forget. When comparing two groups, the runs are often "paired", e.g. by seed or permutation value. When evaluating, we might need the ability to match them up, e.g.,

• for coloring: red if worse than all in all seeds, green if better than all in all seeds
• in ipet: compute number of "consistent" wins/losses, i.e., better/worse than reference run in all seeds

I am not sure that must be stored in the data base, maybe it can also be specified or detected at evaluation time.

[fschloesser]

I am unsure if this is still relevant, since the current setup works quite well.

[ambros-gleixner]

I think it is still relevant, but we should discuss again what exactly we need before this is implemented.