Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_automated_recovery_from_failed_nodes_reactive_IPI.py fails during rack/zone allocation on node addition or during finalizer #10774

Open
am-agrawa opened this issue Oct 29, 2024 · 0 comments · May be fixed by #10769
Assignees
Labels
High Priority High priority issues Squad/Brown

Comments

@am-agrawa
Copy link
Contributor

am-agrawa commented Oct 29, 2024

on ODF 4.17

2024-10-25 19:18:32 tests/functional/z_cluster/nodes/test_automated_recovery_from_failed_nodes_reactive_IPI.py:320:
2024-10-25 19:18:32 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2024-10-25 19:18:32 ocs_ci/ocs/node.py:2523: in wait_for_nodes_racks_or_zones
2024-10-25 19:18:32 for nodes_racks_or_zones in TimeoutSampler(
2024-10-25 19:18:32 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2024-10-25 19:18:32
2024-10-25 19:18:32 self = <ocs_ci.utility.utils.TimeoutSampler object at 0x7f2f62720520>
2024-10-25 19:18:32
2024-10-25 19:18:32 def iter(self):
2024-10-25 19:18:32 if self.start_time is None:
2024-10-25 19:18:32 self.start_time = time.time()
2024-10-25 19:18:32 while True:
2024-10-25 19:18:32 self.last_sample_time = time.time()
2024-10-25 19:18:32 if self.timeout <= (self.last_sample_time - self.start_time):
2024-10-25 19:18:32 raise self.timeout_exc_cls(*self.timeout_exc_args)
2024-10-25 19:18:32 try:
2024-10-25 19:18:32 yield self.func(*self.func_args, **self.func_kwargs)
2024-10-25 19:18:32 except Exception as ex:
2024-10-25 19:18:32 msg = f"Exception raised during iteration: {ex}"
2024-10-25 19:18:32 log.exception(msg)
2024-10-25 19:18:32 if self.timeout <= (time.time() - self.start_time):
2024-10-25 19:18:32 > raise self.timeout_exc_cls(*self.timeout_exc_args)
2024-10-25 19:18:32 E ocs_ci.ocs.exceptions.TimeoutExpiredError: Timed out after 120s running get_nodes_racks_or_zones("failure_domain"="rack", "node_names"=['amagrawa-auto-rec-p7f9m-worker-0-s9sl4'])
2024-10-25 19:18:32
2024-10-25 19:18:32 ocs_ci/utility/utils.py:1459: TimeoutExpiredError

<==========================or==========================>

def finalizer():
if self.extra_node:
nodes.terminate_nodes([self.osd_worker_node], wait=True)
log.info(
f"Successfully terminated node : "
f"{self.osd_worker_node.name} instance"
)
else:
is_recovered = recover_node_to_ready_state(self.osd_worker_node)
if not is_recovered:
log.warning(
f"The recovery of the osd worker node "

          f"{self.osd_worker_node.name} failed. Adding a new OCS worker node..."
        )

E AttributeError: 'NoneType' object has no attribute 'name'

tests/functional/z_cluster/nodes/test_automated_recovery_from_failed_nodes_reactive_IPI.py:246: AttributeError

@am-agrawa am-agrawa added High Priority High priority issues Squad/Brown labels Oct 29, 2024
@am-agrawa am-agrawa self-assigned this Oct 29, 2024
@am-agrawa am-agrawa changed the title test_automated_recovery_from_failed_nodes_reactive_IPI.py fails during rack/zone allocation on node addition test_automated_recovery_from_failed_nodes_reactive_IPI.py fails during rack/zone allocation on node addition or during finalizer Oct 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
High Priority High priority issues Squad/Brown
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant