Added Independent Finding Execution Behavior #1030

andrecsilva · 2025-03-24T12:00:20Z

Overview

Adds independent finding behavior for codemods

Description

Independent finding behavior means that codemods will execute once for each finding (i.e. remediation), as opposed to once for all findings (i.e. hardening). Hardening aims to produce a file with all the changes, while remediation will produce diffs for each finding. Remediation is the default behavior accessible with the codemodder command, while hardening is now accessible as codemodder_hardening.

Additional Details

Remediation won't write any files as the default behavior, this includes the dependencies.
While find and fix codemods will still be accessible over the remediation behavior, there is no difference over the hardening behavior. The only exception being codemods that use semgrep scans for findings.
Most tests with 2+ changes are now being tested for remediation behavior. I've found this gives us a healthy balance of testing for both behaviors while avoiding duplicating each test.

drdavella

Overall this looks good. My comments boil down to two basic issues:

We should preserve the current hardening behavior as the default for now
I'm not totally convinced that locations are being handled correctly for the hardening case

drdavella · 2025-03-25T16:45:22Z

src/codemodder/codemodder.py

@@ -135,6 +136,7 @@ def run(
    sast_only: bool = False,
    ai_client: bool = True,
    log_matched_files: bool = False,
+    hardening: bool = False,


Suggested change

hardening: bool = False,

hardening: bool = True,

drdavella · 2025-03-26T19:42:19Z

.github/workflows/codemod_pygoat.yml

@@ -38,6 +38,6 @@ jobs:
          repository: pixee/pygoat
          path: pygoat
      - name: Run Codemodder
-        run: codemodder --dry-run --output output.codetf pygoat
+        run: codemodder_hardening --dry-run --output output.codetf pygoat


I think we should preserve the existing behavior of the command line for now.

drdavella · 2025-03-26T19:42:50Z

integration_tests/test_dependency_manager.py

@@ -136,7 +136,7 @@ def test_fail_to_add(self, tmp_repo):
        os.chmod(tmp_repo / self.requirements_file, 0o400)

        command = [
-            "codemodder",
+            "codemodder_hardening",


Let's keep the existing behavior for now.

drdavella · 2025-03-26T19:43:17Z

integration_tests/test_multiple_codemods.py

@@ -25,7 +25,7 @@ def test_two_codemods(self, codemods, tmpdir):
        shutil.copy(pathlib.Path(SAMPLES_DIR) / source_file_name, directory)

        command = [
-            "codemodder",
+            "codemodder_hardening",


Preserve existing behavior for now.

drdavella · 2025-03-26T19:44:35Z

pyproject.toml

@@ -45,6 +45,7 @@ Repository = "https://github.com/pixee/codemodder-python"

 [project.scripts]
 codemodder = "codemodder.codemodder:main"
+codemodder_hardening = "codemodder.codemodder:harden"


I don't think this is necessary. We expect per-finding mode to be accessed exclusively by client library calls for now and we should preserve the existing codemodder behavior as the default in the near term.

We do use the new command line script for integration tests.

drdavella · 2025-03-26T19:45:41Z

src/codemodder/codemodder.py

@@ -231,7 +233,7 @@ def run(
    return codetf, 0, token_usage


-def _run_cli(original_args) -> int:
+def _run_cli(original_args, hardening=False) -> int:


Suggested change

def _run_cli(original_args, hardening=False) -> int:

def _run_cli(original_args, hardening=True) -> int:

drdavella · 2025-03-26T19:45:59Z

src/codemodder/codemodder.py

    )
    return status


+def harden():


I don't think this is necessary; see comments above.

drdavella · 2025-03-26T19:46:40Z

src/codemodder/codemods/base_codemod.py

-            contexts.extend([process_file(file) for file in files_to_analyze])
-        else:
+        # Do each result independently and outputs the diffs
+        if not hardening:


This if/else feels like each branch should maybe be a separate function. It would probably enable the call to be a one-liner as well.

drdavella · 2025-03-26T20:16:21Z

src/codemodder/codemods/base_codemod.py

+                    singleton = results.__class__()
+                    singleton.add_result(result)
+                    result_locations = self.get_files_to_analyze(context, singleton)
+                    # We do an execution for each location in the result


I don't think this is correct. We only need to execute each fix per finding. A finding may report multiple locations, but will still result in only a single fix (which itself may touch multiple locations). I think the distinction is subtle but we still execute each codemod only once per finding.

Let me know whether there's something I'm missing or whether maybe the comment isn't quite right.

Two things to note: our transformers are file-based, that is, they execute per file. ChangeSet objects only reports changes to a single file.

This particular piece of code mimics the behavior we had before. If you only had one result with multiple locations, you would run the codemod for each location and produce a changeset object for each location.

What changed is that now each changeset is only associated with a single finding.

clavedeluna

a few code comments that seem unnecessarily explaining code (LLM generated likely) that are worth removing

clavedeluna · 2025-03-26T16:20:10Z

pyproject.toml

@@ -45,6 +45,7 @@ Repository = "https://github.com/pixee/codemodder-python"

 [project.scripts]
 codemodder = "codemodder.codemodder:main"
+codemodder_hardening = "codemodder.codemodder:harden"


for consistency with other commands, I suggest codemodder-hardening

clavedeluna · 2025-03-26T22:54:02Z

src/codemodder/codemods/test/utils.py

@@ -72,9 +73,10 @@ def run_and_assert(

        path_exclude = [f"{tmp_file_path}:{line}" for line in lines_to_exclude or []]

+        print(expected_diff_per_change)


Suggested change

print(expected_diff_per_change)

sonarqubecloud · 2025-04-03T12:31:36Z

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
2.0% Duplication on New Code

See analysis details on SonarQube Cloud

andrecsilva requested review from drdavella and clavedeluna as code owners March 24, 2025 12:00

andrecsilva marked this pull request as draft March 24, 2025 12:00

andrecsilva marked this pull request as ready for review March 25, 2025 12:19

drdavella requested changes Mar 26, 2025

View reviewed changes

clavedeluna reviewed Mar 26, 2025

View reviewed changes

andrecsilva force-pushed the independent-execution branch from e943f3c to 5c37c17 Compare March 28, 2025 13:25

andrecsilva requested review from clavedeluna and drdavella April 2, 2025 16:26

andrecsilva added 21 commits April 3, 2025 09:29

Codemods will now execute once for each finding independently

94aa5da

Adjusted tests to account for individual changes diff

383764c

Fixed a few more unit tests

c66929d

Fixed a few more tests

d812511

Fixed more unit tests

f60975b

Fixed more unit tests

1994a98

Added hardening script

4cc088d

Added integration tests for new behavior

2bbfd1f

Added file rewritten check

a89c0bf

Fixed some integration tests

5cf82fa

More integration tests converted

43b2315

Fixed a few more tests

e5a4126

Remove leftover debugging code

ed02d19

Fixed broken dependency and some docstrings

a0646ab

Fixed pygoat workflow file

9455b35

Fixed some intermittent tests

2d6dbef

Some refactoring

b854bd8

More refactoring

8d2c87c

Small refactoring

f14a005

Reverted default behavior to hardening

192234c

Fixed pygoat test

147b421

andrecsilva added 3 commits April 3, 2025 09:29

Refactored hardening and remediation behavior

05bffa0

Fixed skipping logic

9e4918d

Fixed asserts in integration tests with sonar issues

aceebc2

andrecsilva force-pushed the independent-execution branch from 042b18c to aceebc2 Compare April 3, 2025 12:29

Removed debugging code

7135a34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added Independent Finding Execution Behavior #1030

Added Independent Finding Execution Behavior #1030

andrecsilva commented Mar 24, 2025

drdavella left a comment

drdavella Mar 25, 2025

drdavella Mar 26, 2025

drdavella Mar 26, 2025

drdavella Mar 26, 2025

drdavella Mar 26, 2025

andrecsilva Mar 28, 2025

drdavella Mar 26, 2025

drdavella Mar 26, 2025

drdavella Mar 26, 2025

drdavella Mar 26, 2025

andrecsilva Mar 28, 2025

clavedeluna left a comment

clavedeluna Mar 26, 2025

clavedeluna Mar 26, 2025

sonarqubecloud bot commented Apr 3, 2025

	def _run_cli(original_args, hardening=False) -> int:
	def _run_cli(original_args, hardening=True) -> int:

		@@ -72,9 +73,10 @@ def run_and_assert(

		path_exclude = [f"{tmp_file_path}:{line}" for line in lines_to_exclude or []]

		print(expected_diff_per_change)

Added Independent Finding Execution Behavior #1030

Are you sure you want to change the base?

Added Independent Finding Execution Behavior #1030

Conversation

andrecsilva commented Mar 24, 2025

Overview

Description

Additional Details

drdavella left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clavedeluna left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Apr 3, 2025

Quality Gate passed