-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #14 from chuckwolber/v1.0-patch-cleanup
V1.0 patch cleanup
- Loading branch information
Showing
4 changed files
with
160 additions
and
104 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -3,19 +3,23 @@ From: Wentao Zhang <[email protected]> | |
| Date: Sun, 18 Aug 2024 18:50:22 -0500 | ||
| Subject: [RFC PATCH 0/3] Enable measuring the kernel's Source-based Code Coverage and MC/DC with Clang | ||
|
|
||
| This patch series adds support for building x86-64 kernels with Clang's Source- | ||
| based Code Coverage and measuring modified condition/decision coverage (MC/DC). | ||
| This series adds support for building x86-64 kernels with Clang's Source- | ||
| based Code Coverage[1] in order to facilitate Modified Condition/Decision | ||
| Coverage (MC/DC)[2] that provably correlates to source code for all levels | ||
| of compiler optimization. | ||
|
|
||
| The newly added kernel/llvm-cov/ directory complements the existing gcov | ||
| implementation. gcov works at a lower level and may better reflect the actual | ||
| execution of object code. However, it lacks the necessary information to connect | ||
| coverage measurement back to source code locations and gcov reports are | ||
| sometimes confusing. With a nonzero optimization level (which is the default | ||
| when building the kernel), it's even harder to rebuild the mapping between | ||
| coverage reports and source code. In the following example from | ||
| drivers/firmware/dmi_scan.c, an expression with four leaf conditions are | ||
| reported to have six branch outcomes, which is not ideally informative in many | ||
| use cases. | ||
| implementation. Gcov works at the object code level which may better | ||
| reflect actual execution. However, Gcov lacks the necessary information to | ||
| correlate coverage measurement with source code location when compiler | ||
| optimization level is non-zero (which is the default when building the | ||
| kernel). In addition, gcov reports are occasionally ambiguous when | ||
| attempting to compare with source code level developer intent. | ||
|
|
||
| In the following gcov example from drivers/firmware/dmi_scan.c, an | ||
| expression with four conditions is reported to have six branch outcomes, | ||
| which is not ideally informative in many safety related use cases, such as | ||
| automotive, medical, and aerospace. | ||
|
|
||
| 5: 1068: if (s == e || *e != '/' || !month || month > 12) { | ||
| branch 0 taken 5 (fallthrough) | ||
|
|
@@ -25,12 +29,12 @@ branch 3 taken 0 | |
| branch 4 taken 0 (fallthrough) | ||
| branch 5 taken 5 | ||
|
|
||
| On the other hand, Clang's Source-based Code Coverage [1] does instrumentation | ||
| at the compiler frontend and maintains accurate mapping from coverage measure to | ||
| source code locations. The generated reports can reflect exactly how the code is | ||
| written regardless of optimization and can present advanced metrics like branch | ||
| coverage and MC/DC in a more human-friendly way. Coverage report for the same | ||
| snippet by llvm-cov would look like the below: | ||
| On the other hand, Clang's Source-based Code Coverage instruments at the | ||
| compiler frontend which maintains an accurate mapping from coverage | ||
| measurement to source code location. Coverage reports reflect exactly how | ||
| the code is written regardless of optimization and can present advanced | ||
| metrics like branch coverage and MC/DC in a clearer way. Coverage report | ||
| for the same snippet by llvm-cov would look as follows: | ||
|
|
||
| 1068| 5| if (s == e || *e != '/' || !month || month > 12) { | ||
| ------------------ | ||
|
|
@@ -40,11 +44,12 @@ snippet by llvm-cov would look like the below: | |
| | Branch (1068:39): [True: 0, False: 5] | ||
| ------------------ | ||
|
|
||
| Clang has also added MC/DC support since its release 18.1.0. MC/DC is a fine- | ||
| Clang has added MC/DC support as of its 18.1.0 release. MC/DC is a fine- | ||
| grained coverage metric required by many automotive and aviation industrial | ||
| standards for certifying mission-critical software [2]. In the following example | ||
| from arch/x86/events/probe.c, llvm-cov gives the MC/DC measure for the decision | ||
| at line 43. | ||
| standards for certifying mission-critical software [3]. | ||
|
|
||
| In the following example from arch/x86/events/probe.c, llvm-cov gives the | ||
| MC/DC measurement for the compound logic decision at line 43. | ||
|
|
||
| 43| 12| if (msr[bit].test && !msr[bit].test(bit, data)) | ||
| ------------------ | ||
|
|
@@ -67,49 +72,73 @@ at line 43. | |
| ------------------ | ||
| 44| 5| continue; | ||
|
|
||
| As the results suggest, during the given span of measurement, only condition C2 | ||
| (!msr[bit].test(bit, data)) is covered. That means C2 was evaluated to both true | ||
| and false and in those test vectors C2 affected the decision outcome | ||
| independently. Therefore MC/DC for the shown decision is 1 out of 2 (50.00%). | ||
| As the results suggest, during the span of measurement, only condition C2 | ||
| (!msr[bit].test(bit, data)) is covered. That means C2 was evaluated to both | ||
| true and false, and in those test vectors C2 affected the decision outcome | ||
| independently. Therefore MC/DC for this decision is 1 out of 2 (50.00%). | ||
|
|
||
| To do a full kernel measurement, instrument the kernel with LLVM_COV_KERNEL_MCDC | ||
| enabled, run the testsuites, and collect the raw profile data under | ||
| To do a full kernel measurement, instrument the kernel with | ||
| LLVM_COV_KERNEL_MCDC enabled, and optionally set a | ||
| LLVM_COV_KERNEL_MCDC_MAX_CONDITIONS value (the default is six). Run the | ||
| testsuites, and collect the raw profile data under | ||
| /sys/kernel/debug/llvm-cov/profraw. Such raw profile data can be merged and | ||
| indexed, and used for generating coverage reports in various formats. | ||
|
|
||
| $ cp /sys/kernel/debug/llvm-cov/profraw vmlinux.profraw | ||
| $ llvm-profdata merge vmlinux.profraw -o vmlinux.profdata | ||
| $ llvm-cov show --show-mcdc --show-mcdc-summary \ | ||
| --format=text --use-color=false -output-dir=coverage_reports \ | ||
| -instr-profile vmlinux.profdata vmlinux | ||
|
|
||
| The first patch in the series enables Clang's Source-based Code Coverage for the | ||
| kernel. The second patch disables instrumenting the same set of files that were | ||
| skipped by kernel/gcov/ as well. The third patch enables MC/DC instrumentation | ||
| which is built on top of Source-based Code Coverage. | ||
|
|
||
| This work reuses a portion of code from a previous effort by Sami Tolvanen et | ||
| al. [3], but we aim for source-based *code coverage* required for high assurance | ||
| (MC/DC) while [3] focused more on performance optimization. | ||
|
|
||
| This initial submission is restricted to x86-64. Support for other architectures | ||
| would need a bit more Makefile & linker script modification. Informally we've | ||
| confirmed that arm64 works and more are being tested. | ||
|
|
||
| Note that Source-based Code Coverage is Clang-specific and isn't compatible with | ||
| Clang's gcov support in kernel/gcov/. Currently, kernel/gcov/ is not able to | ||
| measure MC/DC without modifying CFLAGS_GCOV and it would face the same issues in | ||
| terms of source correlation as gcov in general does. | ||
|
|
||
| Some demo and results can be found in [4]. We will talk about this patch series | ||
| in the Refereed Track of LPC 2024 [5] and would really like to hear the | ||
| $ llvm-cov show --show-mcdc --show-mcdc-summary \ | ||
| --format=text --use-color=false -output-dir=coverage_reports \ | ||
| -instr-profile vmlinux.profdata vmlinux | ||
|
|
||
| The first patch in this series enables Clang's Source-based Code Coverage | ||
| in the kernel. The second patch disables instrumentation in the same areas | ||
| that were disabled for kernel/gcov/. The third patch adds kernel | ||
| configuration directives to enable MC/DC instrumentation. | ||
|
|
||
| This work reuses code from a previous effort by Sami Tolvanen et al. [4]. | ||
| Our aim is for source-based *code coverage* required for high assurance | ||
| (MC/DC) while [4] focused more on performance optimization. | ||
|
|
||
| This initial submission is restricted to x86-64. Support for other | ||
| architectures would need a bit more Makefile & linker script modification. | ||
| Informally we've confirmed that arm64 works and more are being tested. | ||
|
|
||
| Note that Source-based Code Coverage is Clang-specific and isn't compatible | ||
| with Clang's gcov support in kernel/gcov/. Currently, kernel/gcov/ is not | ||
| able to measure MC/DC without modifying CFLAGS_GCOV and it would face the | ||
| same issues in terms of source correlation as gcov in general does. | ||
|
|
||
| Some demo and results can be found in [5]. We will talk about this patch | ||
| series in the Refereed Track at LPC 2024 [6] and would appreciate the | ||
| community's feedback. | ||
|
|
||
| Known Limitations: | ||
|
|
||
| Kernel code with logical expressions exceeding | ||
| LVM_COV_KERNEL_MCDC_MAX_CONDITIONS will produce a compiler warning. | ||
| Expressions with up to 45 conditions are found in the Linux kernel source | ||
| tree, but 44 seems to be the max value before the build fails due to kernel | ||
| size. As of LLVM 19 the max number of conditions possible is 32767. | ||
|
|
||
| As of LLVM 19, certain expressions are still not covered, and will produce | ||
| build warnings when they are encountered: | ||
|
|
||
| "[...] if a boolean expression is embedded in the nest of another boolean | ||
| expression but separated by a non-logical operator, this is also not | ||
| supported. For example, in x = (a && b && c && func(d && f)), the d && f | ||
| case starts a new boolean expression that is separated from the other | ||
| conditions by the operator func(). When this is encountered, a warning | ||
| will be generated and the boolean expression will not be | ||
| instrumented." [7] | ||
|
|
||
|
|
||
| [1] https://clang.llvm.org/docs/SourceBasedCodeCoverage.html | ||
| [2] https://digital-library.theiet.org/content/journals/10.1049/sej.1994.0025 | ||
| [3] https://lore.kernel.org/lkml/[email protected]/ | ||
| [4] https://github.com/xlab-uiuc/linux-mcdc | ||
| [5] https://lpc.events/event/18/contributions/1718/ | ||
| [2] https://en.wikipedia.org/wiki/Modified_condition%2Fdecision_coverage | ||
| [3] https://digital-library.theiet.org/content/journals/10.1049/sej.1994.0025 | ||
| [4] https://lore.kernel.org/lkml/[email protected]/ | ||
| [5] https://github.com/xlab-uiuc/linux-mcdc | ||
| [6] https://lpc.events/event/18/contributions/1718/ | ||
| [7] https://clang.llvm.org/docs/SourceBasedCodeCoverage.html#mc-dc-instrumentation | ||
|
|
||
| Wentao Zhang (3): | ||
| [RFC PATCH 1/3] llvm-cov: add Clang's Source-based Code Coverage | ||
|
|
@@ -132,20 +161,20 @@ Wentao Zhang (3): | |
| drivers/firmware/efi/libstub/Makefile | 2 + | ||
| include/asm-generic/vmlinux.lds.h | 38 ++++ | ||
| kernel/Makefile | 1 + | ||
| kernel/llvm-cov/Kconfig | 51 ++++++ | ||
| kernel/llvm-cov/Kconfig | 65 +++++++ | ||
| kernel/llvm-cov/Makefile | 5 + | ||
| kernel/llvm-cov/fs.c | 253 ++++++++++++++++++++++++++ | ||
| kernel/llvm-cov/llvm-cov.h | 156 ++++++++++++++++ | ||
| kernel/trace/Makefile | 1 + | ||
| scripts/Makefile.lib | 21 +++ | ||
| scripts/Makefile.modfinal | 1 + | ||
| scripts/mod/modpost.c | 2 + | ||
| 22 files changed, 551 insertions(+) | ||
| 22 files changed, 565 insertions(+) | ||
| create mode 100644 kernel/llvm-cov/Kconfig | ||
| create mode 100644 kernel/llvm-cov/Makefile | ||
| create mode 100644 kernel/llvm-cov/fs.c | ||
| create mode 100644 kernel/llvm-cov/llvm-cov.h | ||
|
|
||
| -- | ||
| 2.34.1 | ||
| -- | ||
| 2.45.2 | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,9 +4,9 @@ Date: Wed, 14 Aug 2024 15:42:58 -0500 | |
| Subject: [RFC PATCH 1/3] llvm-cov: add Clang's Source-based Code Coverage | ||
| support | ||
|
|
||
| This patch implements the debugfs entries for serializing profiles and | ||
| resetting counters/bitmaps and adds Source-based Code Coverage flags and | ||
| kconfig options. | ||
| Implement debugfs entries for serializing profiles and resetting | ||
| counters/bitmaps. Also adds Source-based Code Coverage flags and kconfig | ||
| options. | ||
|
|
||
| This work reuses a portion of code from a previous effort by Sami | ||
| Tolvanen et al. [1], specifically its debugfs interface and the | ||
|
|
@@ -27,20 +27,20 @@ Signed-off-by: Chuck Wolber <[email protected]> | |
| arch/x86/kernel/vmlinux.lds.S | 2 + | ||
| include/asm-generic/vmlinux.lds.h | 38 +++++ | ||
| kernel/Makefile | 1 + | ||
| kernel/llvm-cov/Kconfig | 30 ++++ | ||
| kernel/llvm-cov/Kconfig | 29 ++++ | ||
| kernel/llvm-cov/Makefile | 5 + | ||
| kernel/llvm-cov/fs.c | 253 ++++++++++++++++++++++++++++++ | ||
| kernel/llvm-cov/llvm-cov.h | 156 ++++++++++++++++++ | ||
| scripts/Makefile.lib | 10 ++ | ||
| scripts/mod/modpost.c | 2 + | ||
| 12 files changed, 502 insertions(+) | ||
| 12 files changed, 501 insertions(+) | ||
| create mode 100644 kernel/llvm-cov/Kconfig | ||
| create mode 100644 kernel/llvm-cov/Makefile | ||
| create mode 100644 kernel/llvm-cov/fs.c | ||
| create mode 100644 kernel/llvm-cov/llvm-cov.h | ||
|
|
||
| diff --git a/Makefile b/Makefile | ||
| index 68ebd6d6b..1750a2b7d 100644 | ||
| index 68ebd6d6b444..1750a2b7dfe8 100644 | ||
| --- a/Makefile | ||
| +++ b/Makefile | ||
| @@ -737,6 +737,9 @@ endif # KBUILD_EXTMOD | ||
|
|
@@ -54,7 +54,7 @@ index 68ebd6d6b..1750a2b7d 100644 | |
| ifdef CONFIG_CC_IS_GCC | ||
| CFLAGS_GCOV += -fno-tree-loop-im | ||
| diff --git a/arch/Kconfig b/arch/Kconfig | ||
| index 975dd22a2..0727265f6 100644 | ||
| index 975dd22a2dbd..0727265f677f 100644 | ||
| --- a/arch/Kconfig | ||
| +++ b/arch/Kconfig | ||
| @@ -1601,6 +1601,7 @@ config ARCH_HAS_KERNEL_FPU_SUPPORT | ||
|
|
@@ -66,7 +66,7 @@ index 975dd22a2..0727265f6 100644 | |
| source "scripts/gcc-plugins/Kconfig" | ||
|
|
||
| diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig | ||
| index 007bab9f2..87a793b82 100644 | ||
| index 007bab9f2a0e..87a793b82bab 100644 | ||
| --- a/arch/x86/Kconfig | ||
| +++ b/arch/x86/Kconfig | ||
| @@ -85,6 +85,7 @@ config X86 | ||
|
|
@@ -78,7 +78,7 @@ index 007bab9f2..87a793b82 100644 | |
| select ARCH_HAS_MEM_ENCRYPT | ||
| select ARCH_HAS_MEMBARRIER_SYNC_CORE | ||
| diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S | ||
| index 6e73403e8..904337722 100644 | ||
| index 6e73403e874f..90433772240a 100644 | ||
| --- a/arch/x86/kernel/vmlinux.lds.S | ||
| +++ b/arch/x86/kernel/vmlinux.lds.S | ||
| @@ -191,6 +191,8 @@ SECTIONS | ||
|
|
@@ -91,7 +91,7 @@ index 6e73403e8..904337722 100644 | |
|
|
||
| . = ALIGN(PAGE_SIZE); | ||
| diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h | ||
| index 1ae447931..770ff8469 100644 | ||
| index 1ae44793132a..770ff8469dcd 100644 | ||
| --- a/include/asm-generic/vmlinux.lds.h | ||
| +++ b/include/asm-generic/vmlinux.lds.h | ||
| @@ -334,6 +334,44 @@ | ||
|
|
@@ -140,7 +140,7 @@ index 1ae447931..770ff8469 100644 | |
| STRUCT_ALIGN(); \ | ||
| __dtb_start = .; \ | ||
| diff --git a/kernel/Makefile b/kernel/Makefile | ||
| index 3c13240df..773e6a9ee 100644 | ||
| index 3c13240dfc9f..773e6a9ee00a 100644 | ||
| --- a/kernel/Makefile | ||
| +++ b/kernel/Makefile | ||
| @@ -117,6 +117,7 @@ obj-$(CONFIG_HAVE_STATIC_CALL) += static_call.o | ||
|
|
@@ -153,10 +153,10 @@ index 3c13240df..773e6a9ee 100644 | |
|
|
||
| diff --git a/kernel/llvm-cov/Kconfig b/kernel/llvm-cov/Kconfig | ||
| new file mode 100644 | ||
| index 000000000..b28090ba9 | ||
| index 000000000000..505eba5bd23c | ||
| --- /dev/null | ||
| +++ b/kernel/llvm-cov/Kconfig | ||
| @@ -0,0 +1,30 @@ | ||
| @@ -0,0 +1,29 @@ | ||
| +# SPDX-License-Identifier: GPL-2.0-only | ||
| +menu "Clang's source-based kernel coverage measurement (EXPERIMENTAL)" | ||
| + | ||
|
|
@@ -186,10 +186,9 @@ index 000000000..b28090ba9 | |
| + profile. | ||
| + | ||
| +endmenu | ||
| + | ||
| diff --git a/kernel/llvm-cov/Makefile b/kernel/llvm-cov/Makefile | ||
| new file mode 100644 | ||
| index 000000000..a0f45e05f | ||
| index 000000000000..a0f45e05f8fb | ||
| --- /dev/null | ||
| +++ b/kernel/llvm-cov/Makefile | ||
| @@ -0,0 +1,5 @@ | ||
|
|
@@ -201,7 +200,7 @@ index 000000000..a0f45e05f | |
| \ No newline at end of file | ||
| diff --git a/kernel/llvm-cov/fs.c b/kernel/llvm-cov/fs.c | ||
| new file mode 100644 | ||
| index 000000000..917ca50d0 | ||
| index 000000000000..c56f660a174e | ||
| --- /dev/null | ||
| +++ b/kernel/llvm-cov/fs.c | ||
| @@ -0,0 +1,253 @@ | ||
|
|
@@ -458,10 +457,9 @@ index 000000000..917ca50d0 | |
| + | ||
| +module_init(llvm_cov_init); | ||
| +module_exit(llvm_cov_exit); | ||
| \ No newline at end of file | ||
| diff --git a/kernel/llvm-cov/llvm-cov.h b/kernel/llvm-cov/llvm-cov.h | ||
| new file mode 100644 | ||
| index 000000000..4a91b3c8e | ||
| index 000000000000..d9551a685756 | ||
| --- /dev/null | ||
| +++ b/kernel/llvm-cov/llvm-cov.h | ||
| @@ -0,0 +1,156 @@ | ||
|
|
@@ -621,9 +619,8 @@ index 000000000..4a91b3c8e | |
| +} | ||
| + | ||
| +#endif /* _LLVM_COV_H */ | ||
| \ No newline at end of file | ||
| diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib | ||
| index fe3668dc4..b9ceaee34 100644 | ||
| index fe3668dc4954..b9ceaee34b28 100644 | ||
| --- a/scripts/Makefile.lib | ||
| +++ b/scripts/Makefile.lib | ||
| @@ -158,6 +158,16 @@ _c_flags += $(if $(patsubst n%,, \ | ||
|
|
@@ -644,7 +641,7 @@ index fe3668dc4..b9ceaee34 100644 | |
| # Enable address sanitizer flags for kernel except some files or directories | ||
| # we don't want to check (depends on variables KASAN_SANITIZE_obj.o, KASAN_SANITIZE) | ||
| diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c | ||
| index d16d0ace2..836c2289b 100644 | ||
| index d16d0ace2775..836c2289b8d8 100644 | ||
| --- a/scripts/mod/modpost.c | ||
| +++ b/scripts/mod/modpost.c | ||
| @@ -743,6 +743,8 @@ static const char *const section_white_list[] = | ||
|
|
@@ -657,5 +654,5 @@ index d16d0ace2..836c2289b 100644 | |
| }; | ||
|
|
||
| -- | ||
| 2.34.1 | ||
| 2.45.2 | ||
|
|
||
Oops, something went wrong.