Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conditionally mark the test cfg as a well known cfg #15007

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Conditionally mark the test cfg as a well known cfg #15007

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
208f817
Add preparatory tests for `lib.test = false` check-cfg changes
Urgau Jan 2, 2025
2593b86
Conditionally mark the `test` cfg as a well known cfg
Urgau Jan 2, 2025
b757af9
Add stderr checking for the conditional `test` well known cfg
Urgau Jan 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 14 additions & 5 deletions src/cargo/core/compiler/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1391,17 +1391,26 @@ fn check_cfg_args(unit: &Unit) -> Vec<OsString> {
}
arg_feature.push("))");

// In addition to the package features, we also include the `test` cfg (since
// compiler-team#785, as to be able to someday apply yt conditionaly), as well
// the `docsrs` cfg from the docs.rs service.
// In addition to the package features, we also conditionaly include the `test` cfg
// based on the unit target "test" field (ie `lib.test = false`, `[[bin]] test = false` and
// others).
//
// We include `docsrs` here (in Cargo) instead of rustc, since there is a much closer
// We also include `docsrs` here (in Cargo) instead of rustc, since there is a much closer
// relationship between Cargo and docs.rs than rustc and docs.rs. In particular, all
// users of docs.rs use Cargo, but not all users of rustc (like Rust-for-Linux) use docs.rs.
let arg_extra = if unit.target.tested()
// Benchmarks default to `test = false` but most of them still use the test crate
// and the `#[test]` attribute, so for now always mark `test` as well known for them.
|| unit.target.is_bench()
{
OsString::from("cfg(docsrs,test)")
} else {
OsString::from("cfg(docsrs)")
};

vec![
OsString::from("--check-cfg"),
OsString::from("cfg(docsrs,test)"),
arg_extra,
OsString::from("--check-cfg"),
arg_feature,
]
Expand Down
5 changes: 4 additions & 1 deletion src/doc/src/reference/cargo-targets.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,10 @@ the target name.
### The `test` field

The `test` field indicates whether or not the target is tested by default by
[`cargo test`]. The default is `true` for lib, bins, and tests.
[`cargo test`], and whenever the target is expected to have tests. Warnings
may be reported when tests are unexpected (i.e., `test = false`).

The default is `true` for lib, bins, and tests.

> **Note**: Examples are built by [`cargo test`] by default to ensure they
> continue to compile, but they are not *tested* by default. Setting `test =
Expand Down
187 changes: 187 additions & 0 deletions tests/testsuite/check_cfg.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -317,6 +317,193 @@ fn well_known_names_values_doctest() {
.run();
}

#[cargo_test(nightly, reason = "warning currently only on nightly")]
fn test_false_lib() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.1.0"
edition = "2018"

[lib]
test = false
"#,
)
.file("src/lib.rs", "#[cfg(test)] mod tests {}")
.build();

p.cargo("check -v")
.with_stderr_does_not_contain(x!("rustc" => "cfg" of "docsrs,test"))
.with_stderr_contains(x!("rustc" => "cfg" of "docsrs"))
Urgau marked this conversation as resolved.
Show resolved Hide resolved
.with_stderr_data(str![[r#"
...
[WARNING] unexpected `cfg` condition name: `test`
...

[WARNING] `foo` (lib) generated 1 warning
[FINISHED] `dev` profile [unoptimized + debuginfo] target(s) in [ELAPSED]s

"#]])
.run();

p.cargo("clean").run();
p.cargo("test -v")
.with_stderr_contains(x!("rustc" => "cfg" of "docsrs"))
.with_stderr_data(str![[r#"
...
[WARNING] unexpected `cfg` condition name: `test`
...

[WARNING] `foo` (lib) generated 1 warning
[FINISHED] `test` profile [unoptimized + debuginfo] target(s) in [ELAPSED]s
[DOCTEST] foo
[RUNNING] [..]

"#]])
.run();

p.cargo("clean").run();
p.cargo("test --lib -v")
.with_stderr_contains(x!("rustc" => "cfg" of "docsrs"))
.with_stderr_data(str![[r#"
...
[WARNING] unexpected `cfg` condition name: `test`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If user has specified --lib, doesn't it mean that they requested to test the lib crate so cfg(test) is expected to be there?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What the command line arguments are shouldn't matter, otherwise users would get a different behavior between a "normal" cargo test and one with arguments like --all-targets.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However, lib.test = false doesn't mean there is no test. It, as of today, is defined as "test" won't run by default. cfg(test) is still relevant regardless if a test is in the default test set.

Maybe I should put the question this way: Why is cfg(test) in a test=false crate unexpected?

Copy link
Member Author

@Urgau Urgau Jan 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO having test = false meaning not tested by default compared to not tests at all is unexpected, at least before reading the documentation I would have assumed that it disabled all the testing.

@jplatte made a further argument in his comment:

Hm, maybe. I guess it depends on what the intention of lib.test = false is. I think the vast majority of projects that set it don't have unit tests at all, and if any were to be introduced, getting a warning would be valuable (whether it results in the test author removing the flag or moving the test code into unit tests). I wonder if there are any projects that use lib.test = false while having unit tests, intentionally.

A similar comment was also put in the URLO thread.

Copy link
Member

@weihanglo weihanglo Jan 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for linking the thread here. Yeah I've read that, and agree that lib.test = false largely implies no test in lib crate at all. That is a valid use case and deserves a lint or warning.

However, fields under Cargo targets are mostly about including in or excluding from the default crate set, and are largely affected by the command-line argument. If Cargo has started emitting unexpected warning for Cargo targets, it may also bite users that disabling tests for other reasons. For example, I've seen people setting test=false because some tests are not expected to run by default for daily development, but only on CI or other special environments.

While that kind of scenario I guess is far less than who don't want tests at all, it is still a valid use case aligning with what the current doc describes. Treating unexpected_cfgs as an indirect way of banning tests in a crate doesn't seem to be the best venue. It is more like a hack because there is no lint for really banning tests.

That being said, I am not surprised if Cargo targets settings don't meet people's expectation. It is always a source of confusions1 😞.

Footnotes

  1. I can actually name a lot of issues here. Let's just have a little peek of them:
    * https://github.com/rust-lang/cargo/issues/8338
    * https://github.com/rust-lang/cargo/issues/10936
    * https://github.com/rust-lang/cargo/issues/10958
    * https://github.com/rust-lang/cargo/issues/13668
    * https://github.com/rust-lang/cargo/issues/13437
    * https://github.com/rust-lang/cargo/issues/13828 ↩

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's a somewhat indirect way of achieving the goal, and it would be nice to also warn on bare #[test] not nested under an explicit cfg(test). Clippy has a lint for that, but ironically it only fires when you run cargo clippy -- --test because otherwise these items are stripped out before Clippy can see them. For the same reason, a dedicated unexpected_tests lint would also have to warn on any occurrence of cfg(... test ...), including nested inside complex cfg expressions, just like unexpected_cfgs would after this PR. I'd also expect a warning for #[cfg_attr(... test ..., blah)] in a crate where tests are "unexpected". So I guess unexpected_tests would be identical to unexpected_cfgs plus warning on #[test]?

But it also wouldn't be a bad solution to just treat bare #[test] as if it came with an implied cfg(test) and emit the same diagnostic if cfg(test) is unexpected. I don't know if that's implementable without unduly complicating unexpected_cfgs and/or #[test] expansion, but it seems semantically appropriate: it matches how #[test] is expanded, except for the subtle difference between rustc --test and rustc --cfg test (that nobody should be relying on). Tailored diagnostics that talk about unit tests in particular can be achieved without an entire new lint. The main thing that's lost is the ability to toggle the lint independently of other unexpected_cfgs warnings, but that seems strange when cfg(... test ...) is part of what the lint would warn about.

Copy link
Member Author

@Urgau Urgau Jan 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The unexpected_cfgs lint do trigger on #[test] but only when passed with --test.

$ rustc +nightly --crate-type=lib --test --check-cfg='cfg()' src/lib.rs
warning: unexpected `cfg` condition name: `test`
 --> src/lib.rs:9:5
  |
9 |     #[test]
  |     ^^^^^^^
  |
  = help: expected names are: `clippy`, `debug_assertions`, `doc`, `doctest`, `fmt_debug`, `miri`, `over
flow_checks`, `panic`, `proc_macro`, `relocation_model`, `rustfmt`, `sanitize`, `sanitizer_cfi_generaliz
e_pointers`, `sanitizer_cfi_normalize_integers`, `target_abi`, `target_arch`, `target_endian`, `target_e
nv`, `target_family`, `target_feature`, `target_has_atomic`, `target_has_atomic_equal_alignment`, `target_has_atomic_load_store`, `target_os`, `target_pointer_width`, `target_thread_local`, `target_vendor`, `ub_checks`, `unix`, and `windows`
  = note: using a cfg inside a attribute macro will use the cfgs from the destination crate and not the ones from the defining crate
  = help: try referring to `test` crate for guidance on how handle this unexpected cfg
  = help: to expect this configuration use `--check-cfg=cfg(test)`
  = note: see <https://doc.rust-lang.org/nightly/rustc/check-cfg.html> for more information about checking conditional configuration
  = note: `#[warn(unexpected_cfgs)]` on by default
  = note: this warning originates in the attribute macro `test` (in Nightly builds, run with -Z macro-backtrace for more info)

warning: 1 warning emitted

We can probably make the lint fire without the --test argument.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unless I missed something in catching up on this, I feel like we'd get more out of test = false being consistent, independent of the build-target type. I'm not too concerned about false positives from people who have #[test] but set test = false.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My main point is the meaning of test = false will change after this PR. I don't mind shipping the particular change. Just want to make sure people aware of and feel good with it. And probably need doc update as well.

Copy link
Member Author

@Urgau Urgau Jan 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have adjusted the documentation with the new behavior.

--> src/lib.rs:1:7
...

[WARNING] `foo` (lib test) generated 1 warning
[FINISHED] `test` profile [unoptimized + debuginfo] target(s) in [ELAPSED]s
[RUNNING] `[ROOT]/foo/target/debug/deps/foo-[HASH][EXE]`

"#]])
.run();
}

#[cargo_test(nightly, reason = "warning currently only on nightly")]
fn test_false_bins() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.1.0"
edition = "2018"

[[bin]]
name = "daemon"
test = false
path = "src/deamon.rs"
"#,
)
.file("src/main.rs", "fn main() {}\n#[cfg(test)] mod tests {}")
.file("src/deamon.rs", "fn main() {}\n#[cfg(test)] mod tests {}")
.build();

p.cargo("check -v")
.with_stderr_contains(x!("rustc" => "cfg" of "docsrs,test")) // for foo
.with_stderr_contains(x!("rustc" => "cfg" of "docsrs")) // for deamon
.with_stderr_data(str![[r#"
...
[WARNING] unexpected `cfg` condition name: `test`
...

[WARNING] `foo` (bin "daemon") generated 1 warning
[FINISHED] `dev` profile [unoptimized + debuginfo] target(s) in [ELAPSED]s

"#]])
.run();
}

#[cargo_test(nightly, reason = "warning currently only on nightly")]
fn test_false_examples() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.1.0"
edition = "2018"

[lib]
test = false

[[example]]
name = "daemon"
test = false
path = "src/deamon.rs"
"#,
)
.file("src/lib.rs", "#[cfg(test)] mod tests {}")
.file("src/deamon.rs", "fn main() {}\n#[cfg(test)] mod tests {}")
.build();

p.cargo("check --examples -v")
.with_stderr_does_not_contain(x!("rustc" => "cfg" of "docsrs,test"))
.with_stderr_contains(x!("rustc" => "cfg" of "docsrs"))
.with_stderr_data(str![[r#"
...
[WARNING] unexpected `cfg` condition name: `test`
...

[WARNING] `foo` (lib) generated 1 warning
...
[WARNING] unexpected `cfg` condition name: `test`
...

[WARNING] `foo` (example "daemon") generated 1 warning
[FINISHED] `dev` profile [unoptimized + debuginfo] target(s) in [ELAPSED]s

"#]])
.run();
}

#[cargo_test(
nightly,
reason = "bench is nightly & warning currently only on nightly"
)]
fn test_false_benches() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
edition = "2018"

[[bench]]
name = "ben1"
test = false
path = "benches/ben1.rs"
"#,
)
.file("src/lib.rs", "")
.file(
"benches/ben1.rs",
r#"
#![feature(test)]
extern crate test;
#[bench] fn run1(_ben: &mut test::Bencher) { }
"#,
)
.build();

// Benches always require the `test` cfg, there should be no warning.
p.cargo("bench --bench ben1")
.with_stderr_data(str![[r#"
[COMPILING] foo v0.0.0 ([ROOT]/foo)
[FINISHED] `bench` profile [optimized] target(s) in [ELAPSED]s
[RUNNING] benches/ben1.rs (target/release/deps/ben1-[HASH][EXE])

"#]])
.run();
}

#[cargo_test]
fn features_doc() {
let p = project()
Expand Down
19 changes: 17 additions & 2 deletions tests/testsuite/test.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4191,7 +4191,19 @@ fn test_hint_workspace_virtual() {
.file("a/src/lib.rs", "#[test] fn t1() {}")
.file("b/Cargo.toml", &basic_manifest("b", "0.1.0"))
.file("b/src/lib.rs", "#[test] fn t1() {assert!(false)}")
.file("c/Cargo.toml", &basic_manifest("c", "0.1.0"))
.file(
weihanglo marked this conversation as resolved.
Show resolved Hide resolved
"c/Cargo.toml",
r#"
[package]
name = "c"
version = "0.1.0"
edition = "2015"

[[example]]
name = "ex1"
test = true
"#,
)
.file(
"c/src/lib.rs",
r#"
Expand Down Expand Up @@ -4275,14 +4287,17 @@ fn test_hint_workspace_virtual() {
[ERROR] test failed, to rerun pass `-p c --bin c`
[RUNNING] tests/t1.rs (target/debug/deps/t1-[HASH][EXE])
[ERROR] test failed, to rerun pass `-p c --test t1`
[RUNNING] unittests examples/ex1.rs (target/debug/examples/ex1-[HASH][EXE])
[ERROR] test failed, to rerun pass `-p c --example ex1`
[DOCTEST] a
[DOCTEST] b
[DOCTEST] c
[ERROR] doctest failed, to rerun pass `-p c --doc`
[ERROR] 4 targets failed:
[ERROR] 5 targets failed:
`-p b --lib`
`-p c --bin c`
`-p c --test t1`
`-p c --example ex1`
`-p c --doc`

"#]])
Expand Down
Loading