Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Case Import Payment Validation #35803

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

Case Import Payment Validation #35803

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
16c0438
create feature flag
zandre-eng Feb 19, 2025
4b7401b
validate required fields exist
zandre-eng Feb 19, 2025
80fb505
validate required fields exist for bulk import
zandre-eng Feb 19, 2025
24971cd
validate field values
zandre-eng Feb 19, 2025
926eb08
minor text update
zandre-eng Feb 20, 2025
9da1701
fix translation formatting
zandre-eng Feb 21, 2025
d1d68f2
update ff name
zandre-eng Feb 21, 2025
523002f
update function name
zandre-eng Feb 21, 2025
36a93de
use sets to check for missing fields
zandre-eng Feb 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions corehq/apps/case_importer/const.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,20 @@
# Special case type used to identify when doing a bulk case import
ALL_CASE_TYPE_IMPORT = 'commcare-all-case-types'

# Fields required when importing cases with "payments" case type
MOMO_REQUIRED_PAYMENT_FIELDS = [
'batch_number',
'phone_number',
'email',
'amount',
'currency',
'payee_note',
'payer_message',
'user_or_case_id',
]

MOMO_PAYMENT_CASE_TYPE = 'payment'


class LookupErrors(object):
NotFound, MultipleResults = list(range(2))
21 changes: 20 additions & 1 deletion corehq/apps/case_importer/views.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@
ALL_CASE_TYPE_IMPORT,
MAX_CASE_IMPORTER_COLUMNS,
MAX_CASE_IMPORTER_ROWS,
MOMO_REQUIRED_PAYMENT_FIELDS,
MOMO_PAYMENT_CASE_TYPE,
)
from corehq.apps.case_importer.exceptions import (
CustomImporterError,
Expand Down Expand Up @@ -53,7 +55,7 @@
)
from corehq.apps.users.decorators import require_permission
from corehq.apps.users.models import HqPermissions
from corehq.toggles import DOMAIN_PERMISSIONS_MIRROR
from corehq.toggles import DOMAIN_PERMISSIONS_MIRROR, MOBILE_WORKER_VERIFICATION
from corehq.util.view_utils import absolute_reverse
from corehq.util.workbook_reading import (
SpreadsheetFileExtError,
Expand Down Expand Up @@ -377,6 +379,16 @@ def excel_fields(request, domain):

case_field_specs = [field_spec.to_json() for field_spec in field_specs]

if MOBILE_WORKER_VERIFICATION.enabled(domain) and case_type == MOMO_PAYMENT_CASE_TYPE:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to worry about bulk import?

Seeing at the function documentation

case_type:
        For a bulk import, this will be displayed
        as the special value for bulk case import/export, and the fetching of
        the case types will be handled in the next step

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should accommodate bulk imports too, as noted by this Jira comment.

It is true that a bulk import will have a special case type and so this check will fail, however we handle validation for bulk imports separately within the _create_bulk_configs() function.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so this check will fail

and so the columns wont be verified for a bulk import here but they are verified in _create_bulk_configs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is correct. The config for a bulk import only gets created in excel_commit() and that is where _create_bulk_configs() is called which will then validate for a bulk import.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does that catch missing columns as well or just checks for values that should not be set?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The check in _create_bulk_configs() will only validate that all the required headers are included. When checking for the values themselves, this will happen in the same place as a single case type import within do_import.py::_validate_payment_fields()

missing_fields = _check_payment_fields_exist(columns)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: since the function is return missing fields, I'd suggest renaming the function to _get_missing_payment_fields

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed in 523002f.

if any(missing_fields):
return render_error(
request,
domain,
_('The Excel file is missing one or more required fields for the "{}" case type: {}'.format(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nit: The error message as is was a little surprising to me when it listed out the missing case types since the message didn't sound like it will do that.

How about changing the text to something below (in my mind it reads a little better)?

The Excel file is missing the following required fields for the "{}" case type: {}

(it is very nit, so feel free to ignore if you want, but I thought I'd mention in nonetheless)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your suggestion does make the text feel a little more streamlined and easier to read. Happy to update this.

Addressed in 926eb08.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additionally, I think the correct way to use the translation would be
_('The Excel file ... "{}" case type: {}').format()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mkangia You're absolutely right. Here is a code ref that confirms how the translation should be. I'll update this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed translation formatting in 9da1701.

MOMO_PAYMENT_CASE_TYPE, ', '.join(missing_fields)
))
)
context = {
'case_type': case_type,
'search_column': search_column,
Expand Down Expand Up @@ -517,3 +529,10 @@ def _bulk_case_upload_api(request, domain):
status_url = absolute_reverse('case_importer_upload_status', args=(domain, upload_id))

return json_response({"code": 200, "message": "success", "status_url": status_url})


def _check_payment_fields_exist(columns):
return [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it be better to use sets here to find the missing fields?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean by doing something as follows?

return set(MOMO_REQUIRED_PAYMENT_FIELDS) - set(columns)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From a performance perspective the difference will be neglible for a list this small, but using sets does make the code footprint a tad less and perhaps a bit easier to read. I'll update this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed in 985e235.

field_name for field_name in MOMO_REQUIRED_PAYMENT_FIELDS
if field_name not in columns
]