Consider having transmute also complain if the pointed-to objects are of different sizes

[larsbergstrom]

Today, cast::transmute will complain if the subjects of the transmute are different sizes. However, if those objects are pointers and the pointed-to objects are different sizes, it will not complain.

This was an issue in Servo because we had two structs that were identical before a Rust upgrade, but after a Rust upgrade some smart pointer-related optimizations changed the layout of one of the structs, resulting in a silent memory corruption. The solution was to add a NonZero wrapper to one of the raw pointers within the structs to disable the compiler optimization (servo/servo@f400d79), but this feels fragile.

[jdm]

It wasn't actually memory corruption; it was the presence of NonNull (via the Box) within one of the types contained in the Option that was the target of the transmute. This caused the enum layout optimization code to deduce that it could use the NonNull value as the discriminant for the transmuted Option, which is what caused us to see Some values where the pre-transmute value was None.

[jdm]

For clarity, this is essentially what we had:

struct PreTransmute {
    ptr: *const (),
}

struct PostTransmute {
    ptr: Box<Something>,
}

let some_value: RefCell<Option<PreTransmute>> = RefCell::new(None);
let tmp = some_value.borrow();
let some_other_value: Ref<Option<PostTransmute>> = unsafe { mem::transmute(tmp) };

and some_other_value would now be classified as Some.

[kmcallister]

This seems like a generalization of -W fat-pointer-transmutes, which is currently broken (#19676).

[Thiez]

Since struct layout is undefined, transmuting one type to another is probably a bad idea in the general case, unless they're both #[repr(C)], or when you're transmuting to and from an array of bytes or something. Which does seems like a good usecase that might be negatively affected by the change suggested in this issue.

[steveklabnik]

Triage: no change that I'm aware of.

[steveklabnik]

Triage: no changes I'm aware of

[asquared31415]

This cannot be made an error specifically with raw pointers because it may be okay. The following code is valid (though odd):

let arr: [u8; 4] = [1, 2, 3, 4];
let arr_ptr: *const u8 = arr.as_ptr();
unsafe {
    let u32_ptr: *const u32 = core::mem::transmute::<_, *const u32>(arr_ptr);
    let u32_val = u32_ptr.read_unaligned();
    assert_eq!(u32_val.to_ne_bytes(), arr);
}

Of course, transmuting between pointers should not be done, in favor of .cast or as, and Clippy will tell you this, but it's perfectly valid because the pointer is a pointer to the whole array and can be used to access more than just one u8.

[workingjubilee]

As @asquared31415 noted this cannot be made an error (breaks backcompat, churns too much code) and is a duplicate otherwise with issues like #34249 which recommend linting against transmutes of pointers in a more general fashion.