feat: compiling using msys2 and mingw

[Kreijstal]

Yes, it's not as pretty as msvc, but it's open source tooling :/. I think this satisfies the immediate need, and if there are further changes I think they might be patchable.

Adding compatibility to use windmc.

@dgreatwood This is what I can muster up to. I know it's not pretty, but this is of course a short term solution. If there are big changes in the manifest, you can ping me and I'll try to adapt it, it should be less work. But here you can cross-compile from linux, that ain't bad right? LMK what you think.

I could also hook up a CI for msys2 compilation if desired.
fixes #1276

[Kreijstal]

I'll try to fix CI.

[dgreatwood]

Hi @Kreijstal -

Firstly, thanks so much for this effort! If I may so, I think it is quite promising.

May I try and describe your strategy? Please correct me if I have misunderstood:

1. You have produced pist_winlog.mc, which is an ".mc" version of pist_winlog.man. I guess you did this manually? Or if you used a tool, what did you use - is it something we could automate?
2. In pist_winlog_impl.cc, you create explicit functions (e.g. EventWritePSTCH_DEBUG_NL_AssumeEnabled) which in the mc.exe case are macros in the generated header file pist_winlog.h. Again, I presume you did this more or less by hand, but again LMK if you used a tool.

Initial questions:

1. Don't we need a header file that corresponds to pist_winlog_impl.cc, let's say pist_winlog_impl.h? Or could we make it a pure header file, pist_winlog_impl.h, inlining all the functions since they're all simple I think?
2. And wouldn't pist_winlog_impl.h need to be included in pist_syslog.cc instead of pist_winlog.h (which mc.exe generates) in the non-mc.exe case? I believe pist_winlog.h should not be in the build if mc.exe is not used.

More generally:

1. The script winscripts/gccsetup.ps1 will always find mc.exe and place it on the path, so perhaps we can add a "--nomcexe" option for gccsetup?
2. We could add tests to meson.build so an error is produced if pist_winlog.man is newer than pist_winlog.mc or pist_winlog_impl.h/.cc. WDYT?

Thanks again!

[Kreijstal]

Hi @Kreijstal -

Firstly, thanks so much for this effort! If I may so, I think it is quite promising.

May I try and describe your strategy? Please correct me if I have misunderstood:

1. You have produced pist_winlog.mc, which is an ".mc" version of pist_winlog.man. I guess you did this manually? Or if you used a tool, what did you use - is it something we could automate?
2. In pist_winlog_impl.cc, you create explicit functions (e.g. EventWritePSTCH_DEBUG_NL_AssumeEnabled) which in the mc.exe case are macros in the generated header file pist_winlog.h. Again, I presume you did this more or less by hand, but again LMK if you used a tool.

Initial questions:

1. Don't we need a header file that corresponds to pist_winlog_impl.cc, let's say pist_winlog_impl.h? Or could we make it a pure header file, pist_winlog_impl.h, inlining all the functions since they're all simple I think?
2. And wouldn't pist_winlog_impl.h need to be included in pist_syslog.cc instead of pist_winlog.h (which mc.exe generates) in the non-mc.exe case? I believe pist_winlog.h should not be in the build if mc.exe is not used.

More generally:

1. The script winscripts/gccsetup.ps1 will always find mc.exe and place it on the path, so perhaps we can add a "--nomcexe" option for gccsetup?
2. We could add tests to meson.build so an error is produced if pist_winlog.man is newer than pist_winlog.mc or pist_winlog_impl.h/.cc. WDYT?

Thanks again!

It was kind of manual, was it macros? I guess we can convert those to macros...too, what is the script gccsetup for? If you have mc.exe you should just use that no?

There is no tool that did this work automatically, but it is reasonable to think one could be written... This would aid mimgw operability! With other projects that use manifests as well! So I really like the idea, this was however something more short term..

[Kreijstal]

@dgreatwood I have addresses the changes, I added a --nomcexe option,
But I also have to thank you, because you gave me the idea of writing the header file way shorter, and way more mantainable I think.

I also wrote a kind of convoluted step, to check the file age, it requires bash, however, not sure if this is undesirable. WDYT?

[Kreijstal]

I managed to write a proto man to mc generator (some things are hardcoded)

import xml.etree.ElementTree as ET

# Define the namespaces
namespaces = {
    'ns': 'http://schemas.microsoft.com/win/2004/08/events',
    'win': 'http://manifests.microsoft.com/win/2004/08/windows/events',
    'xs': 'http://www.w3.org/2001/XMLSchema'
}

# Parse the .man file
tree = ET.parse('pist_winlog.man')
root = tree.getroot()

# Extract provider information
provider = root.find('.//ns:provider', namespaces)
if provider is None:
    print("Error: Could not find the <provider> element in the XML.")
    sys.exit(1)

provider_name = provider.get('name')
provider_guid = provider.get('guid')
provider_symbol = provider.get('symbol')

# Convert provider name to a valid function name (e.g., "Pistache-Provider" -> "Pistache_Provider")
provider_func_name = provider_name.replace('-', '_')

# Extract events
events = provider.find('ns:events', namespaces).findall('ns:event', namespaces)

# Extract localization strings
strings = {}
string_table = root.find('.//ns:stringTable', namespaces).findall('ns:string', namespaces)
for string in string_table:
    strings[string.get('id')] = string.get('value')

# Start building the .mc content
mc_content = ';// SPDX-FileCopyrightText: 2024 Duncan Greatwood\n'
mc_content += ';// SPDX-License-Identifier: Apache-2.0\n\n'
mc_content += 'MessageIdTypedef=DWORD\n\n'

# Provider Information
mc_content += f';// Provider Information\n'
mc_content += f';// Name: {provider_name}\n'
mc_content += f';// GUID: {provider_guid}\n'
mc_content += f';// Symbol: {provider_symbol}\n\n'

# Events
mc_content += ';// Events\n'
for event in events:
    message_id = event.get('value')
    symbolic_name = event.get('symbol')
    message = event.get('message')
    if message.startswith('$(string.Event.'):
        message_id_str = message.lstrip('$(string.Event.').rstrip(')')
        message_text = strings.get(message_id_str, '')
    else:
        message_text = message
    mc_content += f'MessageId={message_id}\n'
    mc_content += f'SymbolicName={symbolic_name}\n'
    mc_content += 'Language=English\n'
    mc_content += f'{message_text}\n.\n\n'
    # Add comments for channel, level, task, and template
    channel = event.get('channel')
    level = event.get('level')
    task = event.get('task')
    template = event.get('template')
    mc_content += f';// Channel: {channel}\n'
    mc_content += f';// Level: {level}\n'
    mc_content += f';// Task: {task}\n'
    mc_content += f';// Template: {template}\n\n'

# Generate Event Descriptors
mc_content += ';// Event Descriptors\n'
for event in events:
    message_id = event.get('value')
    symbolic_name = event.get('symbol')
    mc_content += f';static const EVENT_DESCRIPTOR EventDesc_{symbolic_name} = {{ {message_id}, 1, 0, 0, 0, 0, 0 }};\n'
mc_content += '\n'

# Generate Provider Handle and GUID
mc_content += ';// Provider Handle and GUID\n'
mc_content += f';static REGHANDLE {provider_func_name}Handle = 0;\n'

# Format the GUID correctly
guid_parts = provider_guid.strip('{}').split('-')
guid_hex = [
    f'0x{guid_parts[0]}',  # Data1
    f'0x{guid_parts[1]}',  # Data2
    f'0x{guid_parts[2]}',  # Data3
    f'0x{guid_parts[3][0:2]}',  # Data4[0]
    f'0x{guid_parts[3][2:4]}',  # Data4[1]
    f'0x{guid_parts[4][0:2]}',  # Data4[2]
    f'0x{guid_parts[4][2:4]}',  # Data4[3]
    f'0x{guid_parts[4][4:6]}',  # Data4[4]
    f'0x{guid_parts[4][6:8]}',  # Data4[5]
    f'0x{guid_parts[4][8:10]}',  # Data4[6]
    f'0x{guid_parts[4][10:12]}'  # Data4[7]
]
mc_content += f';static const GUID {provider_symbol} =\n'
mc_content += f';{{ {guid_hex[0]}, {guid_hex[1]}, {guid_hex[2]}, {{ {guid_hex[3]}, {guid_hex[4]}, {guid_hex[5]}, {guid_hex[6]}, {guid_hex[7]}, {guid_hex[8]}, {guid_hex[9]}, {guid_hex[10]} }} }};\n\n'

# Generate Provider Registration Code
mc_content += ';// Provider Registration and Unregistration\n'
mc_content += f';static inline ULONG EventRegister{provider_func_name}() {{\n'
mc_content += f';    return EventRegister(&{provider_symbol}, nullptr, nullptr, &{provider_func_name}Handle);\n'
mc_content += ';}\n\n'
mc_content += f';static inline ULONG EventUnregister{provider_func_name}() {{\n'
mc_content += f';    return EventUnregister({provider_func_name}Handle);\n'
mc_content += ';}\n\n'

# Generate Event Writing Functions
mc_content += ';// Event Writing Functions\n'
mc_content += ';#define GENERATE_EVENT_WRITE_FUNCTION(event_name, event_descriptor) \\\n'
mc_content += ';    static inline ULONG event_name(PCWSTR message) { \\\n'
mc_content += ';        EVENT_DATA_DESCRIPTOR descriptor; \\\n'
mc_content += ';        EventDataDescCreate(&descriptor, message, (ULONG)((wcslen(message) + 1) * sizeof(WCHAR))); \\\n'
mc_content += f';        return EventWrite({provider_func_name}Handle, &event_descriptor, 1, &descriptor); \\\n'
mc_content += ';    }\n\n'

# Generate AssumeEnabled Macros
mc_content += ';// AssumeEnabled Macros\n'
mc_content += ';#define GENERATE_ASSUME_ENABLED_MACRO(event_name) \\\n'
mc_content += ';    static inline ULONG event_name##_AssumeEnabled(PCWSTR message) { \\\n'
mc_content += ';        return event_name(message); \\\n'
mc_content += ';    }\n\n'

# Generate Functions for Each Event
for event in events:
    symbolic_name = event.get('symbol')
    mc_content += f';GENERATE_EVENT_WRITE_FUNCTION(EventWrite{symbolic_name}, EventDesc_{symbolic_name})\n'
    mc_content += f';GENERATE_ASSUME_ENABLED_MACRO(EventWrite{symbolic_name})\n\n'

# Write to .mc file
with open('pist_winlog.mc', 'w', encoding='utf-8') as mc_file:
    mc_file.write(mc_content)

[dgreatwood]

...what is the script gccsetup for? If you have mc.exe you should just use that no?

gccsetup.ps1 checks whether gcc is already setup, and puts it on the path / installs it as needed. It also installs other components that are needed such as meson and even Visual Studio, including mc.exe.

I added a --nomcexe option

Great.

I managed to write a proto man to mc generator

Very cool. This does seem like the best approach if practical.

I also wrote a kind of convoluted step, to check the file age, it requires bash, however, not sure if this is undesirable. WDYT?

So far I've avoided using bash scripts in the Windows cases. I have used some PowerShell scripts e.g. src/winlog/installman.ps1, which is invoked on certain conditions from src/meson.build. Perhaps a similar approach can also work here? And/or this is something I could look at myself.

From what I see (forgive me if I'm not looking correctly), you have some changes you're working on which you haven't pushed yet. Which is fine of course. LMK when you're ready / have pushed, and I can look more closely.

Thanks again.

[Kreijstal]

...what is the script gccsetup for? If you have mc.exe you should just use that no?

gccsetup.ps1 checks whether gcc is already setup, and puts it on the path / installs it as needed. It also installs other components that are needed such as meson and even Visual Studio, including mc.exe.

I added a --nomcexe option

Great.

I managed to write a proto man to mc generator

Very cool. This does seem like the best approach if practical.

I also wrote a kind of convoluted step, to check the file age, it requires bash, however, not sure if this is undesirable. WDYT?

So far I've avoided using bash scripts in the Windows cases. I have used some PowerShell scripts e.g. src/winlog/installman.ps1, which is invoked on certain conditions from src/meson.build. Perhaps a similar approach can also work here? And/or this is something I could look at myself.

From what I see (forgive me if I'm not looking correctly), you have some changes you're working on which you haven't pushed yet. Which is fine of course. LMK when you're ready / have pushed, and I can look more closely.

Thanks again.

Powershell is not really portable (Yes I know .net core exist but it is not provided by all linux distros because the compiler is not fully open source (It's not bootstrapable with an open source toolchain)), I think since I have done the python script I'll just generate the .mc file without doing the history check, This should work on most cases. I hope so at least!

[dgreatwood]

since I have done the python script I'll just generate the .mc file without doing the history check

That seems a good approach.

Powershell is not really portable

Sure, but we should probably only be doing the .mc file generation for gcc builds on Windows, and PowerShell is available on Windows at least. But of course if you can generate everything then this is a non-issue as you said.

[Kreijstal]

since I have done the python script I'll just generate the .mc file without doing the history check

That seems a good approach.

Powershell is not really portable

Sure, but we should probably only be doing the .mc file generation for gcc builds on Windows, and PowerShell is available on Windows at least. But of course if you can generate everything then this is a non-issue as you said.

since I have done the python script I'll just generate the .mc file without doing the history check

That seems a good approach.

Powershell is not really portable

Sure, but we should probably only be doing the .mc file generation for gcc builds on Windows, and PowerShell is available on Windows at least. But of course if you can generate everything then this is a non-issue as you said.

I think I'm done.

[dgreatwood]

Very good. I'll plan to try this out tomorrow.

Thanks again!

[dgreatwood]

Hi @Kreijstal -

I've been running your branch here. It builds quite nicely 👍

I do see an issue with logging when running the "no mc.exe" configuration. In that configuration, when I look at the events deposited into the Windows ETW log, the event viewer shows me:

The description for Event ID 102 from source Pistache-Provider cannot be
found. Either the component that raises this event is not installed on your
local computer or the installation is corrupted. You can install or repair the
component on the local computer.

Whereas, when I run the "WITH mc.exe" configuration (but still building with gcc/msys2 otherwise), I see in the log:

Event 102
INFO Pistache exiting

Similarly for all the "Event 1" events - debug log messages - in the "with mc.exe" case I see meaningful messages like:
DEBUG (11 PSTCH) reactor.cc:343 in shutdown(): Locking shutdown_mutex_ (at 0000012cf92fd0a8)

To get the "description cannot be found" results, I did:

.\winscripts\gccsetup.ps1 -nomcexe
.\winscripts\mesbuilddebug.ps1
.\winscripts\mestestdebug.ps1

(You don't actually have to run the all the tests; e.g. cookie_test_3 should be sufficient to generate log entries).

I note that even when the manifest from the "with mc.exe" is installed, Windows Event Viewer still cannot interpret the events from the "no mc.exe" configuration. This suggests to me that the events written to the log are corrupt in the "no mc.exe" configuration, it's not purely a matter of the manifest; but it's possible something else is going on.

IMPORTANT: The file Building on Windows.txt, the Logging section, explains how I captured and viewed these logs.

BTW, please note the pistachelog.dll and manifest are installed automatically by the meson build process (the build process invokes src\winlog\installman.ps1, which copies pistachelog.dll and executes wevtutil).

Could you take a look at this issue?

Thanks much!