3294 s3 log file uploader

[raftmsohani]

Summary of Changes

Provide a brief summary of changes
Pull request closes #3294_

How to Test

List the steps to test the PR
add the following values into your .env file. Please note that I still haven't added these in the docker, so you might have to add these env vars to be able to pull them into docker instance:

AWS_ACCESS_KEY_ID = ""
AWS_SECRET_ACCESS_KEY = ""
AWS_REGION = "us-east-1"
AWS_S3_BUCKET_NAME = "bucket2025" # make sure you create this bucket at least for testing
AWS_S3_LOGS_PREFIX = "logs"

cd tdrs-frontend && docker-compose up --build
cd tdrs-backend && docker-compose up --build

1. Open http://localhost:3000/ and sign in.
2. add a file for parsing
3. look at the AWS S3 bucket and make sure you see the log file.

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

• [insert ACs here]
• lfrohlich and/or adpennington confirmed that ACs are met.

Deliverable 2: Tested Code

• Are all areas of code introduced in this PR meaningfully tested?
  • If this PR introduces backend code changes, are they meaningfully tested?
  • If this PR introduces frontend code changes, are they meaningfully tested?
• Are code coverage minimums met?
  • Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
  • Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

• Are backend code style checks passing on CircleCI?
• Are frontend code style checks passing on CircleCI?
• Are code maintainability principles being followed?

Deliverable 4: Accessible

• Does this PR complete the epic?
• Are links included to any other gov-approved PRs associated with epic?
• Does PR include documentation for Raft's a11y review?
• Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

• Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

• Does this PR provide background for why coding decisions were made?
• If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces dependencies, are their licenses documented?
• Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

• Does the OWASP Scan pass on CircleCI?
• Do manual code review and manual testing detect any new security issues?
• If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

• the purpose of the research
• methods used to conduct the research
• who participated in the research
• what was tested and how
• impact of research on TDP
• (if applicable) final design mockups produced for TDP development

[codecov]

Codecov Report

Attention: Patch coverage is 67.39130% with 15 lines in your changes missing coverage. Please review.

Project coverage is 90.52%. Comparing base (8ffdb0f) to head (7587d0a).

Files with missing lines	Patch %	Lines
tdrs-backend/tdpservice/log_handler.py	61.53%	14 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3487      +/-   ##
===========================================
- Coverage    90.64%   90.52%   -0.13%     
===========================================
  Files          310      311       +1     
  Lines         8908     8952      +44     
  Branches       677      680       +3     
===========================================
+ Hits          8075     8104      +29     
- Misses         704      718      +14     
- Partials       129      130       +1     

Flag	Coverage Δ
dev-backend	90.30% <67.39%> (-0.14%)	⬇️
dev-frontend	92.07% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
tdrs-backend/tdpservice/parsers/parse.py	82.63% <100.00%> (-0.07%)	⬇️
tdrs-backend/tdpservice/settings/common.py	99.36% <100.00%> (+0.02%)	⬆️
tdrs-backend/tdpservice/log_handler.py	61.53% <61.53%> (ø)

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8ffdb0f...7587d0a. Read the comment docs.

[elipe17]

I keep getting errors trying to submit a datafile:

web-1                | [2025-02-27 16:23:15,274: ERROR/ForkPoolWorker-1] Task tdpservice.scheduling.parser_task.parse[77bc6ffd-3403-4305-8ba5-66d37d721399] raised unexpected: TypeError('expected string or bytes-like object')
web-1                | Traceback (most recent call last):
web-1                |   File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 451, in trace_task
web-1                |     R = retval = fun(*args, **kwargs)
web-1                |   File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 734, in __protected_call__
web-1                |     return self.run(*args, **kwargs)
web-1                |   File "/tdpapp/tdpservice/scheduling/parser_task.py", line 106, in parse
web-1                |     logger.handlers[0].doRollover()
web-1                |   File "/tdpapp/tdpservice/log_handler.py", line 55, in doRollover
web-1                |     self.s3_client.upload_file(
web-1                |   File "/usr/local/lib/python3.10/site-packages/boto3/s3/inject.py", line 143, in upload_file
web-1                |     return transfer.upload_file(
web-1                |   File "/usr/local/lib/python3.10/site-packages/boto3/s3/transfer.py", line 288, in upload_file
web-1                |     future = self._manager.upload(
web-1                |   File "/usr/local/lib/python3.10/site-packages/s3transfer/manager.py", line 322, in upload
web-1                |     self._validate_if_bucket_supported(bucket)
web-1                |   File "/usr/local/lib/python3.10/site-packages/s3transfer/manager.py", line 490, in _validate_if_bucket_supported
web-1                |     match = pattern.match(bucket)
web-1                | TypeError: expected string or bytes-like object

Also, where are you creating the bucket in localstack and overriding the default aws region and access/secret keys?

[elipe17]

see comment