dev: Added package management with Poetry, Ruff for code static analysis

[fukusuket]

What Changed

Ruff (made with Rust), a Python code static analysis tool, looked good, so I decided to try it out :)

Add the following tools for the development environment.

• Python package version management
  • Poetry (like Cargo)
• Python code static analysis
  • Ruff(like Clippy)

This PR does not affect script logic. (This is a PR for improving the development environment)

Evidence

Test Environment

• OS: macOS Sonoma version 14.0
• Hard: MacBook Air(M1, 2020) , Memory 8GB, Core 8
• Poetry: version 1.7.1

Test(README)

Complete the script as follows according to the steps in the README.

sigmac % poetry install --no-root
Creating virtualenv hayabusa_rule_converter-6ygZgvup-py3.12 in /Users/fukusuke/Library/Caches/pypoetry/virtualenvs
Installing dependencies from lock file

Package operations: 3 installs, 0 updates, 0 removals

  • Installing pyyaml (6.0.1)
  • Installing oyaml (1.0)
  • Installing ruff (0.1.13)
sigmac % poetry run python logsource_mapping.py -r ../../../sigma -o ./converted_sigma_rules
[INFO:logsource_mapping.py:462] Start to logsource mapping sigma rules.
[INFO:logsource_mapping.py:499] Loading logsource mapping yaml(sysmon/windows-audit/windows-services) done.
[INFO:logsource_mapping.py:440] Start to collect target yml files path.
...
[INFO:logsource_mapping.py:520] [4111] files created successfully. Created files were saved under [./converted_sigma_rules].
[INFO:logsource_mapping.py:521] Script took [11.11] seconds.
sigmac % 

Test(Code static analysis)

I was able to execute the Ruff command and check the code static analysis results.

sigmac % poetry run ruff logsource_mapping.py                                               
logsource_mapping.py:432:18: F541 [*] f-string without any placeholders
logsource_mapping.py:440:17: F541 [*] f-string without any placeholders
logsource_mapping.py:499:17: F541 [*] f-string without any placeholders
Found 3 errors.
[*] 3 fixable with the `--fix` option.
sigmac % 

I would appreciate it if you could review when you have time🙏

[YamatoSecurity]

@fukusuket Thanks for this!
This is not a problem with this particular PR but I noticed that when a rule is not able to be converted it sometimes does not print the file name:

[ERROR:logsource_mapping.py:299] This rule has incompatible field.{'process_creation': {'EventID': 4688, 'Channel': 'Security'}, 'selection': {'OriginalFileName': 'RUNDLL32.EXE'}, 'filter': {'NewProcessName|endswith': '\\rundll32.exe'}, 'condition': 'selection and not filter'}. skip conversion.
[ERROR:logsource_mapping.py:299] This rule has incompatible field.{'process_creation': {'EventID': 4688, 'Channel': 'Security'}, 'selection': {'Description': 'Execute processes remotely', 'Product': 'Sysinternals PsExec'}, 'filter': {'NewProcessName|endswith': ['\\PsExec.exe', '\\PsExec64.exe']}, 'condition': 'selection and not filter'}. skip conversion.
[ERROR:logsource_mapping.py:299] This rule has incompatible field.{'process_creation': {'EventID': 4688, 'Channel': 'Security'}, 'exec_selection': {'NewProcessName|endswith': '\\rclone.exe', 'ParentProcessName|endswith': ['\\PowerShell.exe', '\\cmd.exe']}, 'command_selection': {'CommandLine|contains': [' pass ', ' user ', ' copy ', ' mega ', ' sync ', ' config ', ' lsd ', ' remote ', ' ls ']}, 'description_selection': {'Description': 'Rsync for cloud storage'}, 'condition': 'command_selection and ( description_selection or exec_selection )'}. skip conversion.

It would be nice to figure out what rules are not being able to get converted. Can you add the filename to the error output?

[hitenkoku]

LGTM

[fukusuket]

@YamatoSecurity @hitenkoku
Thank you so much for checking :)
I added the path of the yml file to the ERROR log as shown below!

[ERROR:logsource_mapping.py:299] Error while converting rule [../../../sigma/deprecated/windows/proc_creation_win_renamed_rundll32.yml]: This rule has incompatible field.{'process_creation': {'EventID': 4688, 'Channel': 'Security'}, 'selection': {'OriginalFileName': 'RUNDLL32.EXE'}, 'filter': {'NewProcessName|endswith': '\\rundll32.exe'}, 'condition': 'selection and not filter'}. skip conversion.
[ERROR:logsource_mapping.py:299] Error while converting rule [../../../sigma/deprecated/windows/proc_creation_win_renamed_psexec.yml]: This rule has incompatible field.{'process_creation': {'EventID': 4688, 'Channel': 'Security'}, 'selection': {'Description': 'Execute processes remotely', 'Product': 'Sysinternals PsExec'}, 'filter': {'NewProcessName|endswith': ['\\PsExec.exe', '\\PsExec64.exe']}, 'condition': 'selection and not filter'}. skip conversion.
[ERROR:logsource_mapping.py:299] Error while converting rule [../../../sigma/deprecated/windows/win_susp_rclone_exec.yml]: This rule has incompatible field.{'process_creation': {'EventID': 4688, 'Channel': 'Security'}, 'exec_selection': {'NewProcessName|endswith': '\\rclone.exe', 'ParentProcessName|endswith': ['\\PowerShell.exe', '\\cmd.exe']}, 'command_selection': {'CommandLine|contains': [' pass ', ' user ', ' copy ', ' mega ', ' sync ', ' config ', ' lsd ', ' remote ', ' ls ']}, 'description_selection': {'Description': 'Rsync for cloud storage'}, 'condition': 'command_selection and ( description_selection or exec_selection )'}. skip conversion.
[INFO:logsource_mapping.py:520] [4111] files created successfully. Created files were saved under [./converted_sigma_rules].
[INFO:logsource_mapping.py:521] Script took [11.03] seconds.

[YamatoSecurity]

@fukusuket Thanks! Looks good now. I updated the English changelog. Could you update the Japanese one for me?

[fukusuket]

@YamatoSecurity
Thank you so much for updating change log :) I updated the Japanese change log!

[YamatoSecurity]

@fukusuket Thank you so much! All looks good, so I will merge this then.