Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Case Import Payment Validation #35803

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
+105 −7
Open

Case Import Payment Validation #35803

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
16c0438
create feature flag
zandre-eng Feb 19, 2025
4b7401b
validate required fields exist
zandre-eng Feb 19, 2025
80fb505
validate required fields exist for bulk import
zandre-eng Feb 19, 2025
24971cd
validate field values
zandre-eng Feb 19, 2025
926eb08
minor text update
zandre-eng Feb 20, 2025
9da1701
fix translation formatting
zandre-eng Feb 21, 2025
d1d68f2
update ff name
zandre-eng Feb 21, 2025
523002f
update function name
zandre-eng Feb 21, 2025
36a93de
use sets to check for missing fields
zandre-eng Feb 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions corehq/apps/case_importer/const.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,28 @@
# Special case type used to identify when doing a bulk case import
ALL_CASE_TYPE_IMPORT = 'commcare-all-case-types'

# Fields required when importing cases with "payments" case type
MOMO_REQUIRED_PAYMENT_FIELDS = [
'batch_number',
'phone_number',
'email',
'amount',
'currency',
'payee_note',
'payer_message',
'user_or_case_id',
]

# Fields that must be absent or blank when importing cases with "payments" case type
MOMO_NO_EDIT_PAYMENT_FIELDS = [
'payment_verified',
'payment_submitted',
'payment_timestamp',
'payment_status',
]

MOMO_PAYMENT_CASE_TYPE = 'payment'


class LookupErrors(object):
NotFound, MultipleResults = list(range(2))
41 changes: 40 additions & 1 deletion corehq/apps/case_importer/do_import.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,19 @@
BULK_UPLOAD_DATE_OPENED,
CASE_IMPORT_DATA_DICTIONARY_VALIDATION,
DOMAIN_PERMISSIONS_MIRROR,
MOBILE_WORKER_VERIFICATION,
)
from corehq.util.metrics import metrics_counter, metrics_histogram
from corehq.util.metrics.load_counters import case_load_counter
from corehq.util.soft_assert import soft_assert
from corehq.util.timer import TimingContext

from .const import LookupErrors
from .const import (
LookupErrors,
MOMO_REQUIRED_PAYMENT_FIELDS,
MOMO_NO_EDIT_PAYMENT_FIELDS,
MOMO_PAYMENT_CASE_TYPE,
)
from .exceptions import (
BlankExternalId,
CaseGeneration,
Expand Down Expand Up @@ -169,6 +175,12 @@ def _do_import(self, spreadsheet):
def import_row(self, row_num, raw_row, import_context):
search_id = self._parse_search_id(raw_row)
fields_to_update = self._populate_updated_fields(raw_row)
if (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a reason for this check to be during import and not earlier when uploading file?
is it because this could take really long so we don't want to do it then?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My main motivation for doing it here is that this is the part in the import where we're looping through all the sheet rows, and so we're able to easily see if there are cell values with empty required fields.

Doing this at an earlier point would mean we would have to do an extra loop through all the rows to validate their values which felt unnecessary. Additionally, failing it here simply results in the row not getting imported. This therefore then doesn't block the entire import and other rows can still get imported if they're valid.

self.mobile_worker_verification_ff_enabled
and self.submission_handler.case_type == MOMO_PAYMENT_CASE_TYPE
):
self._validate_payment_fields(fields_to_update)

if self._has_custom_case_import_operations():
fields_to_update = self._perform_custom_case_import_operations(
row_num,
Expand Down Expand Up @@ -207,6 +219,29 @@ def import_row(self, row_num, raw_row, import_context):

self.submission_handler.add_caseblock(RowAndCase(row_num, caseblock))

def _validate_payment_fields(self, fields):
errors = []
for field_name in MOMO_NO_EDIT_PAYMENT_FIELDS:
if fields.get(field_name):
errors.append(CaseRowError(
column_name=field_name,
message=_(
'This field value cannot be set for cases with the '
'"{}" case type.'.format(MOMO_PAYMENT_CASE_TYPE)
),
))
for field_name in MOMO_REQUIRED_PAYMENT_FIELDS:
if field_name not in fields or not fields[field_name]:
errors.append(CaseRowError(
column_name=field_name,
message=_(
'This field requires a value for cases with the '
'"{}" case type.'.format(MOMO_PAYMENT_CASE_TYPE)
),
))
if any(errors):
raise CaseRowErrorList(errors)

def _has_custom_case_import_operations(self):
return any(
extension.should_call_for_domain(self.domain)
Expand All @@ -227,6 +262,10 @@ def _perform_custom_case_import_operations(self, row_num, raw_row, fields_to_upd
def user(self):
return CouchUser.get_by_user_id(self.config.couch_user_id)

@cached_property
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

def mobile_worker_verification_ff_enabled(self):
return MOBILE_WORKER_VERIFICATION.enabled(self.domain)

def _parse_search_id(self, row):
""" Find and convert the search id in an Excel row """

Expand Down
43 changes: 39 additions & 4 deletions corehq/apps/case_importer/views.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@
ALL_CASE_TYPE_IMPORT,
MAX_CASE_IMPORTER_COLUMNS,
MAX_CASE_IMPORTER_ROWS,
MOMO_REQUIRED_PAYMENT_FIELDS,
MOMO_PAYMENT_CASE_TYPE,
)
from corehq.apps.case_importer.exceptions import (
CustomImporterError,
Expand Down Expand Up @@ -53,7 +55,7 @@
)
from corehq.apps.users.decorators import require_permission
from corehq.apps.users.models import HqPermissions
from corehq.toggles import DOMAIN_PERMISSIONS_MIRROR
from corehq.toggles import DOMAIN_PERMISSIONS_MIRROR, MOBILE_WORKER_VERIFICATION
from corehq.util.view_utils import absolute_reverse
from corehq.util.workbook_reading import (
SpreadsheetFileExtError,
Expand Down Expand Up @@ -291,6 +293,8 @@ def _process_excel_mapping(domain, spreadsheet, search_column):
def _create_bulk_configs(domain, request, case_upload):
all_configs = []
worksheet_titles = _get_workbook_sheet_names(case_upload)
mobile_worker_verification_ff_enabled = MOBILE_WORKER_VERIFICATION.enabled(domain)
errors = []
for index, title in enumerate(worksheet_titles):
with case_upload.get_spreadsheet(index) as spreadsheet:
_, excel_fields, _ = _process_excel_mapping(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

double underscores!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah these were changed to double underscores otherwise they overshadow the gettext import which is imported as _.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you think the double underscores don't look right, I can always change these to have variable names instead 🤔

Expand All @@ -299,6 +303,16 @@ def _create_bulk_configs(domain, request, case_upload):
request.POST['search_field']
)
excel_fields = list(set(excel_fields) - set(RESERVED_FIELDS))
if mobile_worker_verification_ff_enabled and title == MOMO_PAYMENT_CASE_TYPE:
missing_fields = _check_payment_fields_exist(excel_fields)
if missing_fields:
errors.append(_(
'Sheet with case type "{}" is missing one or more required '
'fields: {}'.format(
title, ', '.join(missing_fields)
)
))
break
config = importer_util.ImporterConfig.from_dict({
'couch_user_id': request.couch_user._id,
'excel_fields': excel_fields,
Expand All @@ -310,7 +324,7 @@ def _create_bulk_configs(domain, request, case_upload):
'create_new_cases': request.POST['create_new_cases'] == 'True',
})
all_configs.append(config)
return all_configs
return all_configs, errors


@require_POST
Expand Down Expand Up @@ -377,6 +391,16 @@ def excel_fields(request, domain):

case_field_specs = [field_spec.to_json() for field_spec in field_specs]

if MOBILE_WORKER_VERIFICATION.enabled(domain) and case_type == MOMO_PAYMENT_CASE_TYPE:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to worry about bulk import?

Seeing at the function documentation

case_type:
        For a bulk import, this will be displayed
        as the special value for bulk case import/export, and the fetching of
        the case types will be handled in the next step

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should accommodate bulk imports too, as noted by this Jira comment.

It is true that a bulk import will have a special case type and so this check will fail, however we handle validation for bulk imports separately within the _create_bulk_configs() function.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so this check will fail

and so the columns wont be verified for a bulk import here but they are verified in _create_bulk_configs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is correct. The config for a bulk import only gets created in excel_commit() and that is where _create_bulk_configs() is called which will then validate for a bulk import.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does that catch missing columns as well or just checks for values that should not be set?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The check in _create_bulk_configs() will only validate that all the required headers are included. When checking for the values themselves, this will happen in the same place as a single case type import within do_import.py::_validate_payment_fields()

missing_fields = _check_payment_fields_exist(columns)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: since the function is return missing fields, I'd suggest renaming the function to _get_missing_payment_fields

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed in 523002f.

if any(missing_fields):
return render_error(
request,
domain,
_('The Excel file is missing one or more required fields for the "{}" case type: {}'.format(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nit: The error message as is was a little surprising to me when it listed out the missing case types since the message didn't sound like it will do that.

How about changing the text to something below (in my mind it reads a little better)?

The Excel file is missing the following required fields for the "{}" case type: {}

(it is very nit, so feel free to ignore if you want, but I thought I'd mention in nonetheless)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your suggestion does make the text feel a little more streamlined and easier to read. Happy to update this.

Addressed in 926eb08.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additionally, I think the correct way to use the translation would be
_('The Excel file ... "{}" case type: {}').format()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mkangia You're absolutely right. Here is a code ref that confirms how the translation should be. I'll update this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed translation formatting in 9da1701.

MOMO_PAYMENT_CASE_TYPE, ', '.join(missing_fields)
))
)
context = {
'case_type': case_type,
'search_column': search_column,
Expand Down Expand Up @@ -421,7 +445,9 @@ def excel_commit(request, domain):
case_type = request.POST['case_type']
all_configs = []
if case_type == ALL_CASE_TYPE_IMPORT:
all_configs = _create_bulk_configs(domain, request, case_upload)
all_configs, errors = _create_bulk_configs(domain, request, case_upload)
if any(errors):
return render_error(request, domain, ', '.join(errors))
else:
config = importer_util.ImporterConfig.from_request(request)
all_configs = [config]
Expand Down Expand Up @@ -483,7 +509,9 @@ def _bulk_case_upload_api(request, domain):

all_configs = []
if context['is_bulk_import']:
all_configs = _create_bulk_configs(domain, request, case_upload)
all_configs, errors = _create_bulk_configs(domain, request, case_upload)
if any(errors):
raise ImporterError(', '.join(errors))
else:
with case_upload.get_spreadsheet() as spreadsheet:
_, excel_fields, _ = _process_excel_mapping(domain, spreadsheet, search_column)
Expand Down Expand Up @@ -517,3 +545,10 @@ def _bulk_case_upload_api(request, domain):
status_url = absolute_reverse('case_importer_upload_status', args=(domain, upload_id))

return json_response({"code": 200, "message": "success", "status_url": status_url})


def _check_payment_fields_exist(columns):
return [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it be better to use sets here to find the missing fields?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean by doing something as follows?

return set(MOMO_REQUIRED_PAYMENT_FIELDS) - set(columns)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From a performance perspective the difference will be neglible for a list this small, but using sets does make the code footprint a tad less and perhaps a bit easier to read. I'll update this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed in 985e235.

field_name for field_name in MOMO_REQUIRED_PAYMENT_FIELDS
if field_name not in columns
]
7 changes: 7 additions & 0 deletions corehq/toggles/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3010,3 +3010,10 @@ def domain_has_privilege_from_toggle(privilege_slug, domain):
tag=TAG_SOLUTIONS,
namespaces=[NAMESPACE_DOMAIN],
)

MOBILE_WORKER_VERIFICATION = StaticToggle(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: if this is only for MTN I'd suggest renaming this to MTN_MOBILE_WORKER_VERIFICATION OR MTN_USER_VERIFICATION.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed in d1d68f2.

slug='mobile_worker_verification',
label='Enable user verification using MTN Mobile Money',
tag=TAG_SOLUTIONS,
namespaces=[NAMESPACE_DOMAIN],
)
Loading