3205 reparse command refactor #3361

raftmsohani · 2024-12-11T16:04:34Z

Summary of Changes

Pull request closes #3205 _

Moved all the utility functions to utilize.py, moved frontend reparse command to a separate function and cleaned up clean_and_reparse management command.

How to Test

Follow steps below:

task up

Open http://localhost:3000/ and sign in.
Proceed to admin page
make sure you have OFA admin role (to see the reparse command in admin) and go to datafiles list
Perform reparse admin command and watch the logs

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

reparsing from DAC consistent with actions before/after moving utility functions
lfrohlich and/or adpennington confirmed that ACs are met.

Deliverable 2: Tested Code

Are all areas of code introduced in this PR meaningfully tested?
- If this PR introduces backend code changes, are they meaningfully tested?
- If this PR introduces frontend code changes, are they meaningfully tested?
Are code coverage minimums met?
- Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
- Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

Are backend code style checks passing on CircleCI?
Are frontend code style checks passing on CircleCI?
Are code maintainability principles being followed?

Deliverable 4: Accessible

Does this PR complete the epic?
Are links included to any other gov-approved PRs associated with epic?
Does PR include documentation for Raft's a11y review?
Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

Does this PR provide background for why coding decisions were made?
If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
If this PR introduces dependencies, are their licenses documented?
Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

Does the OWASP Scan pass on CircleCI?
Do manual code review and manual testing detect any new security issues?
If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

the purpose of the research
methods used to conduct the research
who participated in the research
what was tested and how
impact of research on TDP
(if applicable) final design mockups produced for TDP development

…TANF-app into 3205-reparse-command-refactor

raftmsohani · 2024-12-19T14:31:57Z

tdrs-backend/docker-compose.yml

@@ -1,6 +1,4 @@
 # Base Docker compose for all environments
-version: "3.4"


this is depreciated

jtimpe

Working as expected! One minor organization comment

jtimpe · 2024-12-20T14:27:17Z

tdrs-backend/tdpservice/search_indexes/reparse.py

+    delete_associated_models,
+    count_total_num_records,
+    calculate_timeout,
+    handle_datafiles,


does it make sense to just have all these functions in the reparse.py file?

I like the idea of that as well

The idea was to have a cleaner reparse file while also we can re-use some of these functions if needed. However, for some of these functions it might make sense to move them to reparse, for example handle_datafile is specific to reparse

elipe17 · 2024-12-20T16:17:39Z

tdrs-backend/tdpservice/search_indexes/reparse.py

+    )
+
+    is_sequential = assert_sequential_execution(log_context)
+    should_exit(not is_sequential)


Should we update this to raise an exception/return an error to the user so that they know why the reparse didn't happen? Even writing to the console would be a start.

Also, can we move this to the very beginning of the function to avoid unnecessary computation if we arent sequential?

elipe17 · 2024-12-20T16:23:51Z

tdrs-backend/tdpservice/search_indexes/reparse.py

+    fiscal_quarter = None
+    fiscal_year = None
+    all_reparse = False
+    new_indices = False


In a future ticket, could we deduce these fields from the selected datafiles?

I agree, I will leave a todo

tdrs-backend/tdpservice/search_indexes/reparse.py

codecov · 2024-12-26T17:56:05Z

Codecov Report

Attention: Patch coverage is 60.14235% with 112 lines in your changes missing coverage. Please review.

Project coverage is 90.66%. Comparing base (f8353dd) to head (df2230b).
Report is 2 commits behind head on develop.

Files with missing lines	Patch %	Lines
tdrs-backend/tdpservice/search_indexes/reparse.py	30.48%	57 Missing ⚠️
tdrs-backend/tdpservice/search_indexes/utils.py	68.15%	45 Missing and 5 partials ⚠️
...h_indexes/management/commands/clean_and_reparse.py	90.00%	4 Missing ⚠️
tdrs-backend/tdpservice/data_files/tasks.py	50.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3361      +/-   ##
===========================================
- Coverage    91.15%   90.66%   -0.50%     
===========================================
  Files          308      310       +2     
  Lines         8832     8899      +67     
  Branches       670      674       +4     
===========================================
+ Hits          8051     8068      +17     
- Misses         654      704      +50     
  Partials       127      127

Flag	Coverage Δ
dev-backend	`90.44% <60.14%> (-0.57%)`	⬇️
dev-frontend	`92.18% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
tdrs-backend/tdpservice/data_files/tasks.py	`73.91% <50.00%> (ø)`
...h_indexes/management/commands/clean_and_reparse.py	`87.83% <90.00%> (+14.66%)`	⬆️
tdrs-backend/tdpservice/search_indexes/utils.py	`68.15% <68.15%> (ø)`
tdrs-backend/tdpservice/search_indexes/reparse.py	`30.48% <30.48%> (ø)`

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f2bdc38...df2230b. Read the comment docs.

raftmsohani · 2025-01-02T19:31:23Z

tdrs-backend/tdpservice/search_indexes/utils.py

+    return True
+
+
+def should_exit(condition):


This should be needed after the management command is removed

raftmsohani · 2025-02-05T14:26:58Z

backup filename is not associated with reparse model id. some concerns about backups being overwritten.

@ADPennington Fixed the association with meta model id, also added datetime at the end of filename. But can definitely remove it. With datetime, we also have a timestamp for the backup filename.

raftmsohani · 2025-02-05T15:01:14Z

@elipe17 @andrew-jameson @jtimpe : I re-requested your review on this since I had changed the way backup filename is assigned. There was an error for some of backup filenames not getting Meta id and add None to backup filename

elipe17 · 2025-02-05T15:58:26Z

tdrs-backend/tdpservice/search_indexes/reparse.py

+        raise Exception(f"Sequential execution required for selected file ids: {selected_file_ids}")
+    meta_model.save()
+    # Backup the Postgres DB
+    backup_file_name += f"_rpv{meta_model.pk}_{datetime.datetime.now().strftime('%d-%M-%Y-%H%M%s')}.pg"


There is an issue with the strftime. I believe it is using minutes for both the minutes and months. Also, can you separate the hours minutes and seconds with colons?

fixed the issue with minutes/month, but it is not possible to add colon since it is setting the filename. However, I did separate them with '-'

elipe17

Quick change needed for backup filename

lhuxraft · 2025-02-11T16:04:58Z

Blocked on QASP review until review of #3440 completes

ADPennington

LGTM 🚀 thanks @raftmsohani

added separate function for reparse and clean

7fb2cde

raftmsohani self-assigned this Dec 13, 2024

raftmsohani added the backend label Dec 13, 2024

raftmsohani and others added 5 commits December 18, 2024 08:35

cleaning clean_and_reparse

259df3e

Update util.py

7e8f776

cleaned reparse command

1cd6e6e

Merge branch '3205-reparse-command-refactor' of github.com:raft-tech/…

5d8a743

…TANF-app into 3205-reparse-command-refactor

linting

be4f375

raftmsohani requested review from andrew-jameson, jtimpe and elipe17 December 19, 2024 14:24

raftmsohani added the raft review This issue is ready for raft review label Dec 19, 2024

Merge branch 'develop' into 3205-reparse-command-refactor

2c54b68

raftmsohani commented Dec 19, 2024

View reviewed changes

jtimpe approved these changes Dec 20, 2024

View reviewed changes

elipe17 reviewed Dec 20, 2024

View reviewed changes

tdrs-backend/tdpservice/search_indexes/reparse.py Outdated Show resolved Hide resolved

elipe17 reviewed Dec 20, 2024

View reviewed changes

tdrs-backend/tdpservice/search_indexes/reparse.py Outdated Show resolved Hide resolved

Merge branch 'develop' into 3205-reparse-command-refactor

73942d5

raftmsohani commented Jan 2, 2025

View reviewed changes

cleaned up reparse file

e80f3de

raftmsohani requested a review from elipe17 January 2, 2025 21:17

Merge branch 'develop' into 3205-reparse-command-refactor

323cc33

andrew-jameson approved these changes Jan 7, 2025

View reviewed changes

cleaned errors

ea0cba7

raftmsohani requested review from ADPennington and removed request for elipe17 January 7, 2025 16:39

raftmsohani added the QASP Review label Jan 7, 2025

Merge branch 'develop' into 3205-reparse-command-refactor

c1b87ae

raftmsohani requested a review from ADPennington February 5, 2025 14:24

raftmsohani requested review from elipe17, andrew-jameson and jtimpe February 5, 2025 14:58

elipe17 reviewed Feb 5, 2025

View reviewed changes

elipe17 requested changes Feb 5, 2025

View reviewed changes

fixed issue with datetime format

b7509da

raftmsohani requested a review from elipe17 February 5, 2025 16:15

elipe17 approved these changes Feb 7, 2025

View reviewed changes

jtimpe approved these changes Feb 10, 2025

View reviewed changes

Merge branch 'develop' into 3205-reparse-command-refactor

e9d2b5f

raftmsohani removed the request for review from andrew-jameson February 10, 2025 15:22

raftmsohani added QASP Review and removed raft review This issue is ready for raft review labels Feb 10, 2025

raftmsohani and others added 2 commits February 13, 2025 10:16

Merge branch 'develop' into 3205-reparse-command-refactor

ebda8b0

Merge branch 'develop' into 3205-reparse-command-refactor

4bc8add

ADPennington added the Deploy with CircleCI-qasp Deploy to https://tdp-frontend-qasp.app.cloud.gov through CircleCI label Feb 18, 2025

Merge branch 'develop' into 3205-reparse-command-refactor

df2230b

ADPennington approved these changes Feb 19, 2025

View reviewed changes

ADPennington added Ready to Merge and removed QASP Review Deploy with CircleCI-qasp Deploy to https://tdp-frontend-qasp.app.cloud.gov through CircleCI labels Feb 19, 2025

raftmsohani merged commit 9f452a3 into develop Feb 19, 2025
17 checks passed

raftmsohani deleted the 3205-reparse-command-refactor branch February 19, 2025 15:14

ADPennington mentioned this pull request Mar 6, 2025

Parser Refactor #3470

Merged

30 tasks

jtimpe mentioned this pull request Mar 6, 2025

Release/v3.7.8 sprint 119+hotfix HHS/TANF-app#567

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

3205 reparse command refactor #3361

3205 reparse command refactor #3361

raftmsohani commented Dec 11, 2024 •

edited by ADPennington

Loading

raftmsohani Dec 19, 2024

jtimpe left a comment

jtimpe Dec 20, 2024

elipe17 Dec 20, 2024

raftmsohani Jan 2, 2025

elipe17 Dec 20, 2024

elipe17 Dec 20, 2024

elipe17 Dec 20, 2024

raftmsohani Jan 2, 2025

codecov bot commented Dec 26, 2024 •

edited

Loading

raftmsohani Jan 2, 2025

raftmsohani commented Feb 5, 2025

raftmsohani commented Feb 5, 2025

elipe17 Feb 5, 2025

raftmsohani Feb 5, 2025

elipe17 left a comment

lhuxraft commented Feb 11, 2025

ADPennington left a comment

		@@ -1,6 +1,4 @@
		# Base Docker compose for all environments
		version: "3.4"

3205 reparse command refactor #3361

3205 reparse command refactor #3361

Conversation

raftmsohani commented Dec 11, 2024 • edited by ADPennington Loading

Summary of Changes

How to Test

Deliverables

Choose a reason for hiding this comment

jtimpe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Dec 26, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

raftmsohani commented Feb 5, 2025

raftmsohani commented Feb 5, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elipe17 left a comment

Choose a reason for hiding this comment

lhuxraft commented Feb 11, 2025

ADPennington left a comment

Choose a reason for hiding this comment

raftmsohani commented Dec 11, 2024 •

edited by ADPennington

Loading

codecov bot commented Dec 26, 2024 •

edited

Loading