Heap Buffer Overflow in smooth2() in cmsgamma.c

[Xuhxx]

Description

A heap buffer overflow vulnerability has been identified in thesmooth2() in cmsgamma.c in lcms2-2.16. The vulnerability occurs due to numerical instability in the calculation of array elements, which can lead to out-of-bounds memory access.

Result of ASAN:

==882101==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000000038 at pc 0x556d323d0b73 bp 0x7fff522736c0 sp 0x7fff522736b8
READ of size 4 at 0x602000000038 thread T0
    #0 0x556d323d0b72 in smooth2 /home/CVE-lcms/project/lcms2.16/src/cmsgamma.c:1173:12
    #1 0x556d323cf72a in cmsSmoothToneCurve /home/CVE-lcms/project/lcms2.16/src/cmsgamma.c:1251:25
    #2 0x556d323c30a0 in main /home/CVE-lcms/harness/harness-1.cc:46:29
    #3 0x7fcfea7b0d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e)
    #4 0x7fcfea7b0e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: 490fef8403240c91833978d494d39e537409b92e)
    #5 0x556d322e86d4 in _start (/home/CVE-lcms/harness/harness-1+0x776d4) (BuildId: f7451d7038c1b1f2)

0x602000000038 is located 0 bytes after 8-byte region [0x602000000030,0x602000000038)
allocated by thread T0 here:
    #0 0x556d323824fe in malloc (/home/CVE-lcms/harness/harness-1+0x1114fe) (BuildId: f7451d7038c1b1f2)
    #1 0x556d323d5a4d in _cmsMallocDefaultFn /home/CVE-lcms/project/lcms2.16/src/cmserr.c:98:20
    #2 0x556d323d590d in _cmsMalloc /home/CVE-lcms/project/lcms2.16/src/cmserr.c:268:12
    #3 0x556d323d5a7a in _cmsMallocZeroDefaultFn /home/CVE-lcms/project/lcms2.16/src/cmserr.c:107:16
    #4 0x556d323d5f51 in _cmsMallocZero /home/CVE-lcms/project/lcms2.16/src/cmserr.c:275:12
    #5 0x556d323d5d8b in _cmsCallocDefaultFn /home/CVE-lcms/project/lcms2.16/src/cmserr.c:161:12
    #6 0x556d323d6147 in _cmsCalloc /home/CVE-lcms/project/lcms2.16/src/cmserr.c:282:12
    #7 0x556d323cf2f4 in cmsSmoothToneCurve /home/CVE-lcms/project/lcms2.16/src/cmsgamma.c:1229:41
    #8 0x556d323c30a0 in main /home/CVE-lcms/harness/harness-1.cc:46:29
    #9 0x7fcfea7b0d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e)
    #10 0x7fcfea7b0e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: 490fef8403240c91833978d494d39e537409b92e)
    #11 0x556d322e86d4 in _start (/home/CVE-lcms/harness/harness-1+0x776d4) (BuildId: f7451d7038c1b1f2)

SUMMARY: AddressSanitizer: heap-buffer-overflow /home/CVE-lcms/project/lcms2.16/src/cmsgamma.c:1173:12 in smooth2
Shadow bytes around the buggy address:
  0x601ffffffd80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x601ffffffe00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x601ffffffe80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x601fffffff00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x601fffffff80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x602000000000: fa fa 02 fa fa fa 00[fa]fa fa 00 fa fa fa 00 fa
  0x602000000080: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x602000000100: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x602000000180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x602000000200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x602000000280: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==882101==ABORTING

Affected Version

lcms2-2.16

POC

Poc-smooth2.zip
You can reproduce the vulnerability with:
./harness input

Analysis

The vulnerability is triggered in the following code sequence:

Little-CMS/src/cmsgamma.c

Lines 1173 to 1174 in 453bafe

	d[2] = w[2] + 5 * lambda - d[1] * c[1] * c[1];
	c[2] = (-4 * lambda - d[1] * c[1] * e[1]) / d[2];

The issue arises when:

1. The calculation of d[2] results in a value very close to zero or zero itself
2. This near-zero value is then used as a divisor in calculating c[2]
3. The division operation with a near-zero divisor produces an extremely large value for c[2]
4. This large value leads to buffer overflow in subsequent array accesses

[mm2]

Please submit source code of the caller program.