rdma: Support early completion of recv() requests

[yexiang-aws]

Description of changes:

Background:
When using LL (Low Latency) or LL128 protocols, NCCL sets the request pointer to NCCL_NET_OPTIONAL_RECV_COMPLETION in irecv() calls. This indicates that the plugin can complete a receiver request early without plugin explicitly polling the CQ to validate data arrival. This is achievable because NCCL itself following LL protocol semantics will validate data arrival by checking the flag bytes.

Plugin Optimization Details:
This change set early completion by default to be enabled when data progress model is FI_PROGRESS_AUTO.

1. Receiver Side:
   1. Marks request completion immediately after CTRL message send completion
   2. Does not wait for RDMA write operation completion
2. Sender Side:
   1. Uses fi_write instead of fi_writedata, to eliminate unnecessary CQ entries on RX side

Requirements:

• Eager msg mode is disabled: eager_max_size == -1. (With the plugin version at the time of this PR, by default, eager mode is disabled)
• Provider must use FI_PROGRESS_AUTO data progress model

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[bwbarrett]

There's something missing in this patch. The receiver needs to specify whether to use early completion or not for each request. There's no change to request header data in this patch, so we're clearly not doing that.

[rauteric]

Looks like a rebase is needed.

[yexiang-aws]

bot:aws:retest

[yexiang-aws]

AWS CI failed due to infra issue on 4_g4dn_ubuntu2204 Jenkins. Other platforms are passing

==================================== ERRORS ====================================
_______ ERROR at setup of test_run_nccl_tests[0x0-base-all_gather_perf] ________
/home/ubuntu/PortaFiducia/tests/conftest.py:526: in aws_ofi_nccl_install_path
    return install_aws_ofi_nccl_plugin("nvidia", mpi=mpi, checkout_target=aws_ofi_nccl_plugin_version, enable_debug=enable_debug, source_repo=testing_config["test_aws_ofi_nccl_repo"])
/home/ubuntu/PortaFiducia/libraries/aws_ofi_nccl/install.py:78: in install_aws_ofi_nccl_plugin
    raise RuntimeError("Source location {} already exists!".format(source_path))
E   RuntimeError: Source location /home/ubuntu/PortaFiducia/build/libraries/aws_ofi_nccl/797/openmpi/4.1.6/source already exists!```