SCANPY-248 Auto detect test code by guillaume-dequenne · Pull Request #318 · SonarSource/sonar-scanner-python

guillaume-dequenne · 2026-05-08T12:24:24Z

No description provided.

hashicorp-vault-sonar-prod · 2026-05-08T12:24:43Z

sonar-review-alpha · 2026-05-08T12:24:53Z

Summary

Adds automatic detection of test file paths from Python tooling configuration.

When sonar.tests is not explicitly set, the scanner now infers test directories from common sources: pyproject.toml (pytest config), pytest.ini, tox.ini, setup.cfg, or filesystem conventions (directories named "tests", "test", "testing"). This reduces reliance on sonar-python's heuristic that silences issues on test-like files.

New CLI flag --sonar-python-test-file-heuristic-disabled lets users disable this heuristic to treat all files as main code regardless of their path. If tooling declares testpaths but all paths are invalid (don't exist), the heuristic is automatically disabled since user intent was expressed.

What reviewers should know

Core Logic: Start with test_paths_loader.py — this is the new auto-detection module. It tries loaders in order (pyproject.toml → pytest.ini → tox.ini → setup.cfg → filesystem). The return value signals both what tests were found AND whether to disable the heuristic if declared paths were invalid.

Integration Point: configuration_loader.py now calls test_paths_loader.load() when sonar.tests is absent and the heuristic isn't disabled. The heuristic-disabling logic checks for an explicit CLI/env/config value before auto-detecting.

Tests: Extensive unit tests in test_test_paths_loader.py (923 lines) use pyfakefs to test all config formats and edge cases. test_auto_detect_tests.py is an integration test against real project structures.

Config Chain: New property SONAR_PYTHON_TEST_FILE_HEURISTIC_DISABLED was added to properties.py and wired through CLI args. Check the property's cli_getter to see how it handles both --sonar-python-test-file-heuristic-disabled and -Dsonar.python.testFileHeuristic.disabled variants.

Watch For: The distinction between "found nothing" (return False, fall through) and "declared but all invalid" (return True, set disable flag). Also check Windows path handling in _existing_paths() — it handles both absolute paths and drive-relative paths.

Generate Walkthrough
Generate Diagram

🗣️ Give feedback

Seppli11

Overall, the cahnge looks good. It's really cool to see that we're finally parsing settings from the pyproject.toml file and co. (Hopefully the python version is next 🤞 )

One thing I'm not sure about is if it makes sense to also have a heuristic based on the folders in the scanner. It seems to create quite a few strange places where, IMO, the outcome isn't that straight forward.

The other, more nitpicky thing, python_project_loader doesn't seem like an appropriate name for this module, given how it is used and how closely tight it usage is to the present of the test property.

Let me know if you want to discuss anything

Seppli11 · 2026-05-12T14:20:06Z

+        # Running python_project_loader unconditionally would emit confusing warnings about
+        # pytest config even when the result would be discarded.
+        if SONAR_TESTS not in resolved_properties:
+            resolved_properties.update(python_project_loader.load(base_dir))


In a more holistic view, applying the properties from the python_project_loader only if sonar.tests is not set doesn't really make sense to me. If we for example extend the python_project_loader to also detect the python version, things become weird.

Maybe it could make sense renaming the module to python_project_test_source_loader or similar (maybe you find a catchier name 😅 )

Seppli11 · 2026-05-12T15:06:21Z

+    return _load_from_ini_file(base_dir, "setup.cfg", _SETUP_CFG_PYTEST_SECTION)
+
+
+def _load_from_filesystem(base_dir: pathlib.Path) -> Optional[str]:


Does it make sense to add this heuristic here as well, given that sonar-python itself has a more extensive set of heuristics.

I think the heuristic from sonar-python is a bit of a crutch in the sense that it will only disable rules and still report metrics based on the FileType, because we can't contradict the scanner for the file type classification.
This will reduce noise but ultimately still produce an "inconsistent" analysis.

Having it here makes the analysis more consistent because the rule execution is in sync with the metrics reporting. It's an argument to use pysonar instead of the generic sonar-scanner-cli because here, the manual setup of sonar.tests is truly optional (assuming what is inferred matches the user expectations).

That is a good point. Maybe it would make sense to only fall back to the heuristic if the user hasn't set anything, rather than we haven't found a valid test folder

In a way I'm wondering if analysis coming from pysonar shouldn't simply disable the heuristic by default?

Maybe that would make the most sense. It would certainly simplify the flow chart of how the heuristics apply.
And, even in the case where they don't specify anything in their respective configuration files, the file based heuristic should still pick up most cases.

Seppli11 · 2026-05-12T15:10:21Z

+        self._wait_for_ce_completion(workdir)
        return process
+
+    def _wait_for_ce_completion(self, workdir: pathlib.Path) -> None:


This method does a lot of automatic checks and fallbacks. Is this really necessary for a test?

Seppli11 · 2026-05-12T15:14:43Z

+        resp.raise_for_status()
+        return resp.json()
+
+    def get_latest_ce_task(self) -> Optional[dict]:


this seems to be never called. Same applies to search_projects and get_project_measures

Seppli11 · 2026-05-12T15:24:43Z

+            testpaths = []
+            """,
+        )
+        result = python_project_loader.load(Path("."))


If we keep the heuristic, this could lead to a weird behavior where the pyrpoject.tol loader returned an empty set, while the file heuristic loader found a folder. This would now override what the user set

Seppli11 · 2026-05-12T15:27:41Z

+        self.fs.create_dir("tests")
+        # nonexistent/ is not on disk — only tests/ should be returned
+        result = python_project_loader.load(Path("."))
+        self.assertEqual(result[SONAR_TESTS], "tests")


A similar issue could appear here, where for whatever reason, all the user specified tests are not valid paths. If the heuristic picks them up, we're overriding what the user explicitly told us.

I made changes so that if we detect a legitimate attempt to set the test, even if invalid, we disable the test detection heuristic downstream.

Seppli11 · 2026-05-12T15:28:11Z

+    def test_load_from_pytest_ini(self):
+        self.fs.create_file(
+            "pytest.ini",
+            contents="[pytest]\ntestpaths = tests integration\n",


nitpick: multiline strings would be nicer

Seppli11

Looks good. I think like this, it makes a lot of sense and seems IMO intuitive. Thanks for implementing this 😁

One thing, it would probably make sense to add the new parameter as a cli option, as well as running the generate_cli_documentation.py tool to regenerate the documentation.

Seppli11 · 2026-05-13T12:33:07Z

 SONAR_WORKING_DIRECTORY: Key = "sonar.working.directory"
 SONAR_SCM_FORCE_RELOAD_ALL: Key = "sonar.scm.forceReloadAll"
 SONAR_MODULES: Key = "sonar.modules"
+SONAR_PYTHON_TEST_FILE_HEURISTIC_DISABLED: Key = "sonar.python.testFileHeuristic.disabled"


This option should also be added to the configuration.cli module as an option, potentially as a switch

sonarqube-next · 2026-05-13T13:19:21Z

Quality Gate passed for 'Python Scanner'

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
0 Dependency risks
98.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

sonar-review-alpha

The or-based CLI getter bug for SONAR_PYTHON_TEST_FILE_HEURISTIC_DISABLED (previously flagged) is still present — False from --no-sonar-python-test-file-heuristic-disabled is silently discarded. That needs fixing before merge.

🗣️ Give feedback

hashicorp-vault-sonar-prod Bot changed the title ~~Auto detect test code~~ SCANPY-248 Auto detect test code May 8, 2026

This comment was marked as resolved.

Sign in to view

This comment was marked as outdated.

Sign in to view

guillaume-dequenne force-pushed the test-code branch from 0cb9ba3 to dd7a4af Compare May 11, 2026 14:12

This comment was marked as resolved.

Sign in to view

guillaume-dequenne force-pushed the test-code branch from c7dee3a to 4150bcc Compare May 11, 2026 14:43

This comment was marked as outdated.

Sign in to view

guillaume-dequenne force-pushed the test-code branch from 4150bcc to f0bd90b Compare May 11, 2026 14:47

This comment was marked as outdated.

Sign in to view

This comment was marked as resolved.

Sign in to view

guillaume-dequenne force-pushed the test-code branch from c31219c to f53f7e5 Compare May 12, 2026 07:58

This comment was marked as resolved.

Sign in to view

guillaume-dequenne force-pushed the test-code branch from 1dff852 to 1cf9e30 Compare May 12, 2026 09:29

This comment was marked as resolved.

Sign in to view

Auto detect test code

f6fc918

guillaume-dequenne force-pushed the test-code branch from 1cf9e30 to 9d12b07 Compare May 12, 2026 09:43

Prevent detection from running when sonar.tests is set

ce9f8f0

guillaume-dequenne force-pushed the test-code branch from 9d12b07 to ce9f8f0 Compare May 12, 2026 09:44

This comment was marked as outdated.

Sign in to view

Fixes

5660446

This comment was marked as outdated.

Sign in to view

Get task by id

e41141b

guillaume-dequenne force-pushed the test-code branch from 170d90e to e41141b Compare May 12, 2026 11:49

This comment was marked as resolved.

Sign in to view

guillaume-dequenne requested a review from Seppli11 May 12, 2026 11:57

Seppli11 requested changes May 12, 2026

View reviewed changes

This comment was marked as resolved.

Sign in to view

guillaume-dequenne force-pushed the test-code branch from 2429d5d to ca83884 Compare May 13, 2026 08:51

This comment was marked as resolved.

Sign in to view

guillaume-dequenne force-pushed the test-code branch from 565dede to c41788d Compare May 13, 2026 09:45

This comment was marked as resolved.

Sign in to view

guillaume-dequenne force-pushed the test-code branch 2 times, most recently from b1b6bf8 to 29971b5 Compare May 13, 2026 11:10

Fix after review

3e0b7bf

guillaume-dequenne force-pushed the test-code branch from 7ec4848 to 3e0b7bf Compare May 13, 2026 11:26

This comment was marked as resolved.

Sign in to view

guillaume-dequenne requested a review from Seppli11 May 13, 2026 11:41

Seppli11 approved these changes May 13, 2026

View reviewed changes

This comment was marked as resolved.

Sign in to view

Further fixes

c22a766

guillaume-dequenne force-pushed the test-code branch from 176edd8 to c22a766 Compare May 13, 2026 13:16

sonar-review-alpha Bot reviewed May 13, 2026

View reviewed changes

guillaume-dequenne merged commit dbea81c into master May 13, 2026
20 checks passed

guillaume-dequenne deleted the test-code branch May 13, 2026 13:29

		return _load_from_ini_file(base_dir, "setup.cfg", _SETUP_CFG_PYTEST_SECTION)


		def _load_from_filesystem(base_dir: pathlib.Path) -> Optional[str]:

Conversation

guillaume-dequenne commented May 8, 2026

Uh oh!

hashicorp-vault-sonar-prod Bot commented May 8, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sonar-review-alpha Bot commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What reviewers should know

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as resolved.

Uh oh!

Seppli11 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Seppli11 May 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

Seppli11 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

hashicorp-vault-sonar-prod Bot commented May 8, 2026 •

edited by atlassian Bot

Loading

sonar-review-alpha Bot commented May 8, 2026 •

edited

Loading

Seppli11 May 13, 2026 •

edited

Loading