-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use-After-Free in BGP ORF usage of prefix lists #18138
Comments
There's now an ASAN report:
|
donaldsharp
added a commit
to donaldsharp/frr
that referenced
this issue
Feb 14, 2025
We are very very rarely seeing this crash: 0 0x7f36ba48e389 in prefix_list_apply_ext lib/plist.c:789 1 0x55eff3fa4126 in subgroup_announce_check bgpd/bgp_route.c:2334 2 0x55eff3fa858e in subgroup_process_announce_selected bgpd/bgp_route.c:3440 3 0x55eff4016488 in subgroup_announce_table bgpd/bgp_updgrp_adv.c:808 4 0x55eff401664e in subgroup_announce_route bgpd/bgp_updgrp_adv.c:861 5 0x55eff40111df in peer_af_announce_route bgpd/bgp_updgrp.c:2223 6 0x55eff3f884cb in bgp_announce_route_timer_expired bgpd/bgp_route.c:5892 7 0x7f36ba4ec239 in event_call lib/event.c:2019 8 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 9 0x55eff3e668b7 in main bgpd/bgp_main.c:557 10 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 11 0x7f36b9e2d304 in __libc_start_main_impl ../csu/libc-start.c:360 12 0x55eff3e64a30 in _start (/home/ci/cibuild.1407/frr-source/bgpd/.libs/bgpd+0x2fda30) 0x608000037038 is located 24 bytes inside of 88-byte region [0x608000037020,0x608000037078) freed by thread T0 here: 0 0x7f36ba8b76a8 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52 1 0x7f36ba439bd7 in qfree lib/memory.c:131 2 0x7f36ba48d3a3 in prefix_list_free lib/plist.c:156 3 0x7f36ba48d3a3 in prefix_list_delete lib/plist.c:247 4 0x7f36ba48fbef in prefix_bgp_orf_remove_all lib/plist.c:1516 5 0x55eff3f679c4 in bgp_route_refresh_receive bgpd/bgp_packet.c:2841 6 0x55eff3f70bab in bgp_process_packet bgpd/bgp_packet.c:4069 7 0x7f36ba4ec239 in event_call lib/event.c:2019 8 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 9 0x55eff3e668b7 in main bgpd/bgp_main.c:557 10 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 previously allocated by thread T0 here: 0 0x7f36ba8b83b7 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:77 1 0x7f36ba4392e4 in qcalloc lib/memory.c:106 2 0x7f36ba48d0de in prefix_list_new lib/plist.c:150 3 0x7f36ba48d0de in prefix_list_insert lib/plist.c:186 4 0x7f36ba48d0de in prefix_list_get lib/plist.c:204 5 0x7f36ba48f9df in prefix_bgp_orf_set lib/plist.c:1479 6 0x55eff3f67ba6 in bgp_route_refresh_receive bgpd/bgp_packet.c:2920 7 0x55eff3f70bab in bgp_process_packet bgpd/bgp_packet.c:4069 8 0x7f36ba4ec239 in event_call lib/event.c:2019 9 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 10 0x55eff3e668b7 in main bgpd/bgp_main.c:557 11 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 Let's just stop trying to save the pointer around in the peer->orf_plist data structure. There are other design problems but at least lets stop the crash from possibly happening. Fixes: FRRouting#18138 Signed-off-by: Donald Sharp <[email protected]>
mergify bot
pushed a commit
that referenced
this issue
Feb 14, 2025
We are very very rarely seeing this crash: 0 0x7f36ba48e389 in prefix_list_apply_ext lib/plist.c:789 1 0x55eff3fa4126 in subgroup_announce_check bgpd/bgp_route.c:2334 2 0x55eff3fa858e in subgroup_process_announce_selected bgpd/bgp_route.c:3440 3 0x55eff4016488 in subgroup_announce_table bgpd/bgp_updgrp_adv.c:808 4 0x55eff401664e in subgroup_announce_route bgpd/bgp_updgrp_adv.c:861 5 0x55eff40111df in peer_af_announce_route bgpd/bgp_updgrp.c:2223 6 0x55eff3f884cb in bgp_announce_route_timer_expired bgpd/bgp_route.c:5892 7 0x7f36ba4ec239 in event_call lib/event.c:2019 8 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 9 0x55eff3e668b7 in main bgpd/bgp_main.c:557 10 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 11 0x7f36b9e2d304 in __libc_start_main_impl ../csu/libc-start.c:360 12 0x55eff3e64a30 in _start (/home/ci/cibuild.1407/frr-source/bgpd/.libs/bgpd+0x2fda30) 0x608000037038 is located 24 bytes inside of 88-byte region [0x608000037020,0x608000037078) freed by thread T0 here: 0 0x7f36ba8b76a8 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52 1 0x7f36ba439bd7 in qfree lib/memory.c:131 2 0x7f36ba48d3a3 in prefix_list_free lib/plist.c:156 3 0x7f36ba48d3a3 in prefix_list_delete lib/plist.c:247 4 0x7f36ba48fbef in prefix_bgp_orf_remove_all lib/plist.c:1516 5 0x55eff3f679c4 in bgp_route_refresh_receive bgpd/bgp_packet.c:2841 6 0x55eff3f70bab in bgp_process_packet bgpd/bgp_packet.c:4069 7 0x7f36ba4ec239 in event_call lib/event.c:2019 8 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 9 0x55eff3e668b7 in main bgpd/bgp_main.c:557 10 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 previously allocated by thread T0 here: 0 0x7f36ba8b83b7 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:77 1 0x7f36ba4392e4 in qcalloc lib/memory.c:106 2 0x7f36ba48d0de in prefix_list_new lib/plist.c:150 3 0x7f36ba48d0de in prefix_list_insert lib/plist.c:186 4 0x7f36ba48d0de in prefix_list_get lib/plist.c:204 5 0x7f36ba48f9df in prefix_bgp_orf_set lib/plist.c:1479 6 0x55eff3f67ba6 in bgp_route_refresh_receive bgpd/bgp_packet.c:2920 7 0x55eff3f70bab in bgp_process_packet bgpd/bgp_packet.c:4069 8 0x7f36ba4ec239 in event_call lib/event.c:2019 9 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 10 0x55eff3e668b7 in main bgpd/bgp_main.c:557 11 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 Let's just stop trying to save the pointer around in the peer->orf_plist data structure. There are other design problems but at least lets stop the crash from possibly happening. Fixes: #18138 Signed-off-by: Donald Sharp <[email protected]> (cherry picked from commit 3d43d7b)
mergify bot
pushed a commit
that referenced
this issue
Feb 14, 2025
We are very very rarely seeing this crash: 0 0x7f36ba48e389 in prefix_list_apply_ext lib/plist.c:789 1 0x55eff3fa4126 in subgroup_announce_check bgpd/bgp_route.c:2334 2 0x55eff3fa858e in subgroup_process_announce_selected bgpd/bgp_route.c:3440 3 0x55eff4016488 in subgroup_announce_table bgpd/bgp_updgrp_adv.c:808 4 0x55eff401664e in subgroup_announce_route bgpd/bgp_updgrp_adv.c:861 5 0x55eff40111df in peer_af_announce_route bgpd/bgp_updgrp.c:2223 6 0x55eff3f884cb in bgp_announce_route_timer_expired bgpd/bgp_route.c:5892 7 0x7f36ba4ec239 in event_call lib/event.c:2019 8 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 9 0x55eff3e668b7 in main bgpd/bgp_main.c:557 10 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 11 0x7f36b9e2d304 in __libc_start_main_impl ../csu/libc-start.c:360 12 0x55eff3e64a30 in _start (/home/ci/cibuild.1407/frr-source/bgpd/.libs/bgpd+0x2fda30) 0x608000037038 is located 24 bytes inside of 88-byte region [0x608000037020,0x608000037078) freed by thread T0 here: 0 0x7f36ba8b76a8 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52 1 0x7f36ba439bd7 in qfree lib/memory.c:131 2 0x7f36ba48d3a3 in prefix_list_free lib/plist.c:156 3 0x7f36ba48d3a3 in prefix_list_delete lib/plist.c:247 4 0x7f36ba48fbef in prefix_bgp_orf_remove_all lib/plist.c:1516 5 0x55eff3f679c4 in bgp_route_refresh_receive bgpd/bgp_packet.c:2841 6 0x55eff3f70bab in bgp_process_packet bgpd/bgp_packet.c:4069 7 0x7f36ba4ec239 in event_call lib/event.c:2019 8 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 9 0x55eff3e668b7 in main bgpd/bgp_main.c:557 10 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 previously allocated by thread T0 here: 0 0x7f36ba8b83b7 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:77 1 0x7f36ba4392e4 in qcalloc lib/memory.c:106 2 0x7f36ba48d0de in prefix_list_new lib/plist.c:150 3 0x7f36ba48d0de in prefix_list_insert lib/plist.c:186 4 0x7f36ba48d0de in prefix_list_get lib/plist.c:204 5 0x7f36ba48f9df in prefix_bgp_orf_set lib/plist.c:1479 6 0x55eff3f67ba6 in bgp_route_refresh_receive bgpd/bgp_packet.c:2920 7 0x55eff3f70bab in bgp_process_packet bgpd/bgp_packet.c:4069 8 0x7f36ba4ec239 in event_call lib/event.c:2019 9 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 10 0x55eff3e668b7 in main bgpd/bgp_main.c:557 11 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 Let's just stop trying to save the pointer around in the peer->orf_plist data structure. There are other design problems but at least lets stop the crash from possibly happening. Fixes: #18138 Signed-off-by: Donald Sharp <[email protected]> (cherry picked from commit 3d43d7b)
mergify bot
pushed a commit
that referenced
this issue
Feb 14, 2025
We are very very rarely seeing this crash: 0 0x7f36ba48e389 in prefix_list_apply_ext lib/plist.c:789 1 0x55eff3fa4126 in subgroup_announce_check bgpd/bgp_route.c:2334 2 0x55eff3fa858e in subgroup_process_announce_selected bgpd/bgp_route.c:3440 3 0x55eff4016488 in subgroup_announce_table bgpd/bgp_updgrp_adv.c:808 4 0x55eff401664e in subgroup_announce_route bgpd/bgp_updgrp_adv.c:861 5 0x55eff40111df in peer_af_announce_route bgpd/bgp_updgrp.c:2223 6 0x55eff3f884cb in bgp_announce_route_timer_expired bgpd/bgp_route.c:5892 7 0x7f36ba4ec239 in event_call lib/event.c:2019 8 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 9 0x55eff3e668b7 in main bgpd/bgp_main.c:557 10 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 11 0x7f36b9e2d304 in __libc_start_main_impl ../csu/libc-start.c:360 12 0x55eff3e64a30 in _start (/home/ci/cibuild.1407/frr-source/bgpd/.libs/bgpd+0x2fda30) 0x608000037038 is located 24 bytes inside of 88-byte region [0x608000037020,0x608000037078) freed by thread T0 here: 0 0x7f36ba8b76a8 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52 1 0x7f36ba439bd7 in qfree lib/memory.c:131 2 0x7f36ba48d3a3 in prefix_list_free lib/plist.c:156 3 0x7f36ba48d3a3 in prefix_list_delete lib/plist.c:247 4 0x7f36ba48fbef in prefix_bgp_orf_remove_all lib/plist.c:1516 5 0x55eff3f679c4 in bgp_route_refresh_receive bgpd/bgp_packet.c:2841 6 0x55eff3f70bab in bgp_process_packet bgpd/bgp_packet.c:4069 7 0x7f36ba4ec239 in event_call lib/event.c:2019 8 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 9 0x55eff3e668b7 in main bgpd/bgp_main.c:557 10 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 previously allocated by thread T0 here: 0 0x7f36ba8b83b7 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:77 1 0x7f36ba4392e4 in qcalloc lib/memory.c:106 2 0x7f36ba48d0de in prefix_list_new lib/plist.c:150 3 0x7f36ba48d0de in prefix_list_insert lib/plist.c:186 4 0x7f36ba48d0de in prefix_list_get lib/plist.c:204 5 0x7f36ba48f9df in prefix_bgp_orf_set lib/plist.c:1479 6 0x55eff3f67ba6 in bgp_route_refresh_receive bgpd/bgp_packet.c:2920 7 0x55eff3f70bab in bgp_process_packet bgpd/bgp_packet.c:4069 8 0x7f36ba4ec239 in event_call lib/event.c:2019 9 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 10 0x55eff3e668b7 in main bgpd/bgp_main.c:557 11 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 Let's just stop trying to save the pointer around in the peer->orf_plist data structure. There are other design problems but at least lets stop the crash from possibly happening. Fixes: #18138 Signed-off-by: Donald Sharp <[email protected]> (cherry picked from commit 3d43d7b)
This was referenced Feb 14, 2025
mergify bot
pushed a commit
that referenced
this issue
Feb 14, 2025
We are very very rarely seeing this crash: 0 0x7f36ba48e389 in prefix_list_apply_ext lib/plist.c:789 1 0x55eff3fa4126 in subgroup_announce_check bgpd/bgp_route.c:2334 2 0x55eff3fa858e in subgroup_process_announce_selected bgpd/bgp_route.c:3440 3 0x55eff4016488 in subgroup_announce_table bgpd/bgp_updgrp_adv.c:808 4 0x55eff401664e in subgroup_announce_route bgpd/bgp_updgrp_adv.c:861 5 0x55eff40111df in peer_af_announce_route bgpd/bgp_updgrp.c:2223 6 0x55eff3f884cb in bgp_announce_route_timer_expired bgpd/bgp_route.c:5892 7 0x7f36ba4ec239 in event_call lib/event.c:2019 8 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 9 0x55eff3e668b7 in main bgpd/bgp_main.c:557 10 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 11 0x7f36b9e2d304 in __libc_start_main_impl ../csu/libc-start.c:360 12 0x55eff3e64a30 in _start (/home/ci/cibuild.1407/frr-source/bgpd/.libs/bgpd+0x2fda30) 0x608000037038 is located 24 bytes inside of 88-byte region [0x608000037020,0x608000037078) freed by thread T0 here: 0 0x7f36ba8b76a8 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52 1 0x7f36ba439bd7 in qfree lib/memory.c:131 2 0x7f36ba48d3a3 in prefix_list_free lib/plist.c:156 3 0x7f36ba48d3a3 in prefix_list_delete lib/plist.c:247 4 0x7f36ba48fbef in prefix_bgp_orf_remove_all lib/plist.c:1516 5 0x55eff3f679c4 in bgp_route_refresh_receive bgpd/bgp_packet.c:2841 6 0x55eff3f70bab in bgp_process_packet bgpd/bgp_packet.c:4069 7 0x7f36ba4ec239 in event_call lib/event.c:2019 8 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 9 0x55eff3e668b7 in main bgpd/bgp_main.c:557 10 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 previously allocated by thread T0 here: 0 0x7f36ba8b83b7 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:77 1 0x7f36ba4392e4 in qcalloc lib/memory.c:106 2 0x7f36ba48d0de in prefix_list_new lib/plist.c:150 3 0x7f36ba48d0de in prefix_list_insert lib/plist.c:186 4 0x7f36ba48d0de in prefix_list_get lib/plist.c:204 5 0x7f36ba48f9df in prefix_bgp_orf_set lib/plist.c:1479 6 0x55eff3f67ba6 in bgp_route_refresh_receive bgpd/bgp_packet.c:2920 7 0x55eff3f70bab in bgp_process_packet bgpd/bgp_packet.c:4069 8 0x7f36ba4ec239 in event_call lib/event.c:2019 9 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 10 0x55eff3e668b7 in main bgpd/bgp_main.c:557 11 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 Let's just stop trying to save the pointer around in the peer->orf_plist data structure. There are other design problems but at least lets stop the crash from possibly happening. Fixes: #18138 Signed-off-by: Donald Sharp <[email protected]> (cherry picked from commit 3d43d7b)
mergify bot
pushed a commit
that referenced
this issue
Feb 14, 2025
We are very very rarely seeing this crash: 0 0x7f36ba48e389 in prefix_list_apply_ext lib/plist.c:789 1 0x55eff3fa4126 in subgroup_announce_check bgpd/bgp_route.c:2334 2 0x55eff3fa858e in subgroup_process_announce_selected bgpd/bgp_route.c:3440 3 0x55eff4016488 in subgroup_announce_table bgpd/bgp_updgrp_adv.c:808 4 0x55eff401664e in subgroup_announce_route bgpd/bgp_updgrp_adv.c:861 5 0x55eff40111df in peer_af_announce_route bgpd/bgp_updgrp.c:2223 6 0x55eff3f884cb in bgp_announce_route_timer_expired bgpd/bgp_route.c:5892 7 0x7f36ba4ec239 in event_call lib/event.c:2019 8 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 9 0x55eff3e668b7 in main bgpd/bgp_main.c:557 10 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 11 0x7f36b9e2d304 in __libc_start_main_impl ../csu/libc-start.c:360 12 0x55eff3e64a30 in _start (/home/ci/cibuild.1407/frr-source/bgpd/.libs/bgpd+0x2fda30) 0x608000037038 is located 24 bytes inside of 88-byte region [0x608000037020,0x608000037078) freed by thread T0 here: 0 0x7f36ba8b76a8 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52 1 0x7f36ba439bd7 in qfree lib/memory.c:131 2 0x7f36ba48d3a3 in prefix_list_free lib/plist.c:156 3 0x7f36ba48d3a3 in prefix_list_delete lib/plist.c:247 4 0x7f36ba48fbef in prefix_bgp_orf_remove_all lib/plist.c:1516 5 0x55eff3f679c4 in bgp_route_refresh_receive bgpd/bgp_packet.c:2841 6 0x55eff3f70bab in bgp_process_packet bgpd/bgp_packet.c:4069 7 0x7f36ba4ec239 in event_call lib/event.c:2019 8 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 9 0x55eff3e668b7 in main bgpd/bgp_main.c:557 10 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 previously allocated by thread T0 here: 0 0x7f36ba8b83b7 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:77 1 0x7f36ba4392e4 in qcalloc lib/memory.c:106 2 0x7f36ba48d0de in prefix_list_new lib/plist.c:150 3 0x7f36ba48d0de in prefix_list_insert lib/plist.c:186 4 0x7f36ba48d0de in prefix_list_get lib/plist.c:204 5 0x7f36ba48f9df in prefix_bgp_orf_set lib/plist.c:1479 6 0x55eff3f67ba6 in bgp_route_refresh_receive bgpd/bgp_packet.c:2920 7 0x55eff3f70bab in bgp_process_packet bgpd/bgp_packet.c:4069 8 0x7f36ba4ec239 in event_call lib/event.c:2019 9 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 10 0x55eff3e668b7 in main bgpd/bgp_main.c:557 11 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 Let's just stop trying to save the pointer around in the peer->orf_plist data structure. There are other design problems but at least lets stop the crash from possibly happening. Fixes: #18138 Signed-off-by: Donald Sharp <[email protected]> (cherry picked from commit 3d43d7b)
mergify bot
pushed a commit
that referenced
this issue
Feb 14, 2025
We are very very rarely seeing this crash: 0 0x7f36ba48e389 in prefix_list_apply_ext lib/plist.c:789 1 0x55eff3fa4126 in subgroup_announce_check bgpd/bgp_route.c:2334 2 0x55eff3fa858e in subgroup_process_announce_selected bgpd/bgp_route.c:3440 3 0x55eff4016488 in subgroup_announce_table bgpd/bgp_updgrp_adv.c:808 4 0x55eff401664e in subgroup_announce_route bgpd/bgp_updgrp_adv.c:861 5 0x55eff40111df in peer_af_announce_route bgpd/bgp_updgrp.c:2223 6 0x55eff3f884cb in bgp_announce_route_timer_expired bgpd/bgp_route.c:5892 7 0x7f36ba4ec239 in event_call lib/event.c:2019 8 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 9 0x55eff3e668b7 in main bgpd/bgp_main.c:557 10 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 11 0x7f36b9e2d304 in __libc_start_main_impl ../csu/libc-start.c:360 12 0x55eff3e64a30 in _start (/home/ci/cibuild.1407/frr-source/bgpd/.libs/bgpd+0x2fda30) 0x608000037038 is located 24 bytes inside of 88-byte region [0x608000037020,0x608000037078) freed by thread T0 here: 0 0x7f36ba8b76a8 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52 1 0x7f36ba439bd7 in qfree lib/memory.c:131 2 0x7f36ba48d3a3 in prefix_list_free lib/plist.c:156 3 0x7f36ba48d3a3 in prefix_list_delete lib/plist.c:247 4 0x7f36ba48fbef in prefix_bgp_orf_remove_all lib/plist.c:1516 5 0x55eff3f679c4 in bgp_route_refresh_receive bgpd/bgp_packet.c:2841 6 0x55eff3f70bab in bgp_process_packet bgpd/bgp_packet.c:4069 7 0x7f36ba4ec239 in event_call lib/event.c:2019 8 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 9 0x55eff3e668b7 in main bgpd/bgp_main.c:557 10 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 previously allocated by thread T0 here: 0 0x7f36ba8b83b7 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:77 1 0x7f36ba4392e4 in qcalloc lib/memory.c:106 2 0x7f36ba48d0de in prefix_list_new lib/plist.c:150 3 0x7f36ba48d0de in prefix_list_insert lib/plist.c:186 4 0x7f36ba48d0de in prefix_list_get lib/plist.c:204 5 0x7f36ba48f9df in prefix_bgp_orf_set lib/plist.c:1479 6 0x55eff3f67ba6 in bgp_route_refresh_receive bgpd/bgp_packet.c:2920 7 0x55eff3f70bab in bgp_process_packet bgpd/bgp_packet.c:4069 8 0x7f36ba4ec239 in event_call lib/event.c:2019 9 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 10 0x55eff3e668b7 in main bgpd/bgp_main.c:557 11 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 Let's just stop trying to save the pointer around in the peer->orf_plist data structure. There are other design problems but at least lets stop the crash from possibly happening. Fixes: #18138 Signed-off-by: Donald Sharp <[email protected]> (cherry picked from commit 3d43d7b)
This was referenced Feb 14, 2025
mergify bot
pushed a commit
that referenced
this issue
Feb 14, 2025
We are very very rarely seeing this crash: 0 0x7f36ba48e389 in prefix_list_apply_ext lib/plist.c:789 1 0x55eff3fa4126 in subgroup_announce_check bgpd/bgp_route.c:2334 2 0x55eff3fa858e in subgroup_process_announce_selected bgpd/bgp_route.c:3440 3 0x55eff4016488 in subgroup_announce_table bgpd/bgp_updgrp_adv.c:808 4 0x55eff401664e in subgroup_announce_route bgpd/bgp_updgrp_adv.c:861 5 0x55eff40111df in peer_af_announce_route bgpd/bgp_updgrp.c:2223 6 0x55eff3f884cb in bgp_announce_route_timer_expired bgpd/bgp_route.c:5892 7 0x7f36ba4ec239 in event_call lib/event.c:2019 8 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 9 0x55eff3e668b7 in main bgpd/bgp_main.c:557 10 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 11 0x7f36b9e2d304 in __libc_start_main_impl ../csu/libc-start.c:360 12 0x55eff3e64a30 in _start (/home/ci/cibuild.1407/frr-source/bgpd/.libs/bgpd+0x2fda30) 0x608000037038 is located 24 bytes inside of 88-byte region [0x608000037020,0x608000037078) freed by thread T0 here: 0 0x7f36ba8b76a8 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52 1 0x7f36ba439bd7 in qfree lib/memory.c:131 2 0x7f36ba48d3a3 in prefix_list_free lib/plist.c:156 3 0x7f36ba48d3a3 in prefix_list_delete lib/plist.c:247 4 0x7f36ba48fbef in prefix_bgp_orf_remove_all lib/plist.c:1516 5 0x55eff3f679c4 in bgp_route_refresh_receive bgpd/bgp_packet.c:2841 6 0x55eff3f70bab in bgp_process_packet bgpd/bgp_packet.c:4069 7 0x7f36ba4ec239 in event_call lib/event.c:2019 8 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 9 0x55eff3e668b7 in main bgpd/bgp_main.c:557 10 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 previously allocated by thread T0 here: 0 0x7f36ba8b83b7 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:77 1 0x7f36ba4392e4 in qcalloc lib/memory.c:106 2 0x7f36ba48d0de in prefix_list_new lib/plist.c:150 3 0x7f36ba48d0de in prefix_list_insert lib/plist.c:186 4 0x7f36ba48d0de in prefix_list_get lib/plist.c:204 5 0x7f36ba48f9df in prefix_bgp_orf_set lib/plist.c:1479 6 0x55eff3f67ba6 in bgp_route_refresh_receive bgpd/bgp_packet.c:2920 7 0x55eff3f70bab in bgp_process_packet bgpd/bgp_packet.c:4069 8 0x7f36ba4ec239 in event_call lib/event.c:2019 9 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 10 0x55eff3e668b7 in main bgpd/bgp_main.c:557 11 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 Let's just stop trying to save the pointer around in the peer->orf_plist data structure. There are other design problems but at least lets stop the crash from possibly happening. Fixes: #18138 Signed-off-by: Donald Sharp <[email protected]> (cherry picked from commit 3d43d7b)
mergify bot
pushed a commit
that referenced
this issue
Feb 14, 2025
We are very very rarely seeing this crash: 0 0x7f36ba48e389 in prefix_list_apply_ext lib/plist.c:789 1 0x55eff3fa4126 in subgroup_announce_check bgpd/bgp_route.c:2334 2 0x55eff3fa858e in subgroup_process_announce_selected bgpd/bgp_route.c:3440 3 0x55eff4016488 in subgroup_announce_table bgpd/bgp_updgrp_adv.c:808 4 0x55eff401664e in subgroup_announce_route bgpd/bgp_updgrp_adv.c:861 5 0x55eff40111df in peer_af_announce_route bgpd/bgp_updgrp.c:2223 6 0x55eff3f884cb in bgp_announce_route_timer_expired bgpd/bgp_route.c:5892 7 0x7f36ba4ec239 in event_call lib/event.c:2019 8 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 9 0x55eff3e668b7 in main bgpd/bgp_main.c:557 10 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 11 0x7f36b9e2d304 in __libc_start_main_impl ../csu/libc-start.c:360 12 0x55eff3e64a30 in _start (/home/ci/cibuild.1407/frr-source/bgpd/.libs/bgpd+0x2fda30) 0x608000037038 is located 24 bytes inside of 88-byte region [0x608000037020,0x608000037078) freed by thread T0 here: 0 0x7f36ba8b76a8 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52 1 0x7f36ba439bd7 in qfree lib/memory.c:131 2 0x7f36ba48d3a3 in prefix_list_free lib/plist.c:156 3 0x7f36ba48d3a3 in prefix_list_delete lib/plist.c:247 4 0x7f36ba48fbef in prefix_bgp_orf_remove_all lib/plist.c:1516 5 0x55eff3f679c4 in bgp_route_refresh_receive bgpd/bgp_packet.c:2841 6 0x55eff3f70bab in bgp_process_packet bgpd/bgp_packet.c:4069 7 0x7f36ba4ec239 in event_call lib/event.c:2019 8 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 9 0x55eff3e668b7 in main bgpd/bgp_main.c:557 10 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 previously allocated by thread T0 here: 0 0x7f36ba8b83b7 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:77 1 0x7f36ba4392e4 in qcalloc lib/memory.c:106 2 0x7f36ba48d0de in prefix_list_new lib/plist.c:150 3 0x7f36ba48d0de in prefix_list_insert lib/plist.c:186 4 0x7f36ba48d0de in prefix_list_get lib/plist.c:204 5 0x7f36ba48f9df in prefix_bgp_orf_set lib/plist.c:1479 6 0x55eff3f67ba6 in bgp_route_refresh_receive bgpd/bgp_packet.c:2920 7 0x55eff3f70bab in bgp_process_packet bgpd/bgp_packet.c:4069 8 0x7f36ba4ec239 in event_call lib/event.c:2019 9 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 10 0x55eff3e668b7 in main bgpd/bgp_main.c:557 11 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 Let's just stop trying to save the pointer around in the peer->orf_plist data structure. There are other design problems but at least lets stop the crash from possibly happening. Fixes: #18138 Signed-off-by: Donald Sharp <[email protected]> (cherry picked from commit 3d43d7b)
eqvinox
pushed a commit
to opensourcerouting/frr
that referenced
this issue
Feb 15, 2025
We are very very rarely seeing this crash: 0 0x7f36ba48e389 in prefix_list_apply_ext lib/plist.c:789 1 0x55eff3fa4126 in subgroup_announce_check bgpd/bgp_route.c:2334 2 0x55eff3fa858e in subgroup_process_announce_selected bgpd/bgp_route.c:3440 3 0x55eff4016488 in subgroup_announce_table bgpd/bgp_updgrp_adv.c:808 4 0x55eff401664e in subgroup_announce_route bgpd/bgp_updgrp_adv.c:861 5 0x55eff40111df in peer_af_announce_route bgpd/bgp_updgrp.c:2223 6 0x55eff3f884cb in bgp_announce_route_timer_expired bgpd/bgp_route.c:5892 7 0x7f36ba4ec239 in event_call lib/event.c:2019 8 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 9 0x55eff3e668b7 in main bgpd/bgp_main.c:557 10 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 11 0x7f36b9e2d304 in __libc_start_main_impl ../csu/libc-start.c:360 12 0x55eff3e64a30 in _start (/home/ci/cibuild.1407/frr-source/bgpd/.libs/bgpd+0x2fda30) 0x608000037038 is located 24 bytes inside of 88-byte region [0x608000037020,0x608000037078) freed by thread T0 here: 0 0x7f36ba8b76a8 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52 1 0x7f36ba439bd7 in qfree lib/memory.c:131 2 0x7f36ba48d3a3 in prefix_list_free lib/plist.c:156 3 0x7f36ba48d3a3 in prefix_list_delete lib/plist.c:247 4 0x7f36ba48fbef in prefix_bgp_orf_remove_all lib/plist.c:1516 5 0x55eff3f679c4 in bgp_route_refresh_receive bgpd/bgp_packet.c:2841 6 0x55eff3f70bab in bgp_process_packet bgpd/bgp_packet.c:4069 7 0x7f36ba4ec239 in event_call lib/event.c:2019 8 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 9 0x55eff3e668b7 in main bgpd/bgp_main.c:557 10 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 previously allocated by thread T0 here: 0 0x7f36ba8b83b7 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:77 1 0x7f36ba4392e4 in qcalloc lib/memory.c:106 2 0x7f36ba48d0de in prefix_list_new lib/plist.c:150 3 0x7f36ba48d0de in prefix_list_insert lib/plist.c:186 4 0x7f36ba48d0de in prefix_list_get lib/plist.c:204 5 0x7f36ba48f9df in prefix_bgp_orf_set lib/plist.c:1479 6 0x55eff3f67ba6 in bgp_route_refresh_receive bgpd/bgp_packet.c:2920 7 0x55eff3f70bab in bgp_process_packet bgpd/bgp_packet.c:4069 8 0x7f36ba4ec239 in event_call lib/event.c:2019 9 0x7f36ba41a22a in frr_run lib/libfrr.c:1295 10 0x55eff3e668b7 in main bgpd/bgp_main.c:557 11 0x7f36b9e2d249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 Let's just stop trying to save the pointer around in the peer->orf_plist data structure. There are other design problems but at least lets stop the crash from possibly happening. Fixes: FRRouting#18138 Signed-off-by: Donald Sharp <[email protected]> (cherry picked from commit 3d43d7b)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We're seeing an intermittent crash in a BGP ORF test:
https://ci1.netdef.org/artifact/TESTING-POTATOUVMPARA/POTATODEB12ARM8/build-1197/Topotato-Report/test_bgp_orf_py__TestBGPORF.html#i4,
Ancillary information:
only seen on ARM8 so farnow seen on AMD64the crash itself seems to be variadic in the sense that it happens roughly in this point of sequence, but not necessarily in the same code locationseems to be consistent after all (might've been poor reporting)This combination of "factlets" is suggestive of a memory synchronization race condition or general memory corruption. ARM has a weaker memory consistency model than x86 TSO. However, it is not immediately obvious to me how the variousbgpd
threads would be involved here to result in this; as I can see it most things should be happening on the main thread really.There is unfortunately no coredump available at this point, I'm working on getting one. I'm also trying to get it to crash under valgrind. Getting it reproduced outside of CI has so far failed.
This issue is opened to get more eyes on this; crashing
bgpd
is a pretty bad bug in any case. It's at minimum an availability/denial of service security issue ifbgpd
can be made to crash, worst case if it's memory corruption it may be exploitable.The test is modeled after
bgp_orf
in topotests, which should in theory show the same issue. I have not attempted to reproduce it on that yet. The topotato test source can be found here: https://github.com/opensourcerouting/topotato/blob/master/test_bgp_orf.py. Feedback on topotato is not invited or welcome on this specific issue.The text was updated successfully, but these errors were encountered: