What should the ABI compatibility rules be for repr($int) enums?

@CAD97

Currently we do not declare that repr($int) enums are ABI-compatible with $int, at least not in the ABI compatibility docs. We do however say or at least heavily imply that it's fine to use a repr($int) enum on the Rust side together with a C enum on the other side whose underlying type matches $int.

The C standard says that such an enum is compatible with $int. However, CFI implementations violate the C standard and do not allow function signatures to type-pun between enums and their underlying $int. IMO that's quite rude of them, standards exist for a reason. But we can't just ignore that fact, or at least if we do we should be aware of the consequences. If we say that in Rust, repr($int) enums are ABI-compatible with $int, then we must give them the same CFI tag, or else CFI would trigger on entirely well-defined Rust code. That would make such an enum not CFI-compatible with the same enum on the C side. (This is why rust-lang/rust#128600 did not make it.)

(The same question also arises for repr(C) enums where the underlying integer type is left implicit. However, there it would probably be less surprising to say that those are not ABI-compatible with that implicit underlying integer type.)

So, what could we do?

Convince clang that integer normalization should also normalize enums to their underlying integer type. (Either for all enums, or only for enums with an explicit underlying type.) We already require integer normalization for CFI to work with Rust, and it seems reasonable to include enums in that, at least those with an explicitly declared underlying integer type.
Declare it EB to do function signature type punning between repr($int) enums and $int. That would at least avoid nasal demons. Miri currently makes this UB and we haven't gotten reports about this, so making them fully ABI-compatible does not seem to be in high demand.
Make repr($int) enums are ABI-compatible with $int unless they have a #[cfi_encoding] attribute, in which case a mismatch becomes EB. That would make such enums not CFI-compatible with the corresponding C enum by default: the enums would get an integer CFI tag by default, and FFI code would have to explicitly declare the CFI encoding to be CFI-compatible with the corresponding C code.
Anything else?

My preferred option is 1. CFI should not interfere with well-defined C programs. But I don't know if there is any chance that clang might do that, or if this would have a serious impact on CFI's ability to prevent exploits. Option 2 also seems acceptable, albeit somewhat unfortunate.
Cc @CAD97 @Jules-Bertholet @Darksonn @rcvalle

(1) and (3) are not mutually exclusive. We could convince clang to normalize their enums by default, but also provide an annotation to opt out of that. Users who feel that not doing normalization is important to prevent exploits can choose to opt out their enums.

One idea:

Make #[repr(u32)] enums ABI compatible with u32, and make #[repr(C, u32)] enums ABI compatible with a C enum whose backing integer type is u32.

Anyways, I think we could probably have CFI unify all enums with their backing integer type. I don't think it would affect CFI's strength in a super meaningful way.

Though do keep in mind that such modifications to CFI are generally new flags that modify CFI, rather than just changing the base CFI abi.

Make #[repr(u32)] enums ABI compatible with u32, and make #[repr(C, u32)] enums ABI compatible with a C enum whose backing integer type is u32.

That seems to be a deprecated combination -- this code emits a FCW:

#[repr(C, u8)]
enum Enum8 {
    Variant0,
    Variant1,
}

Though do keep in mind that such modifications to CFI are generally new flags that modify CFI, rather than just changing the base CFI abi.

As long as we say that those flags are mandatory for CFI in Rust, that does not seem to be a problem. "To mix C and Rust code under CFI, the C code must be compiled with the following flags: ...".

@kupiakos

That seems to be a deprecated combination -- this code emits a FCW:
#[repr(C, u8)]
enum Enum8 {
    Variant0,
    Variant1,
}

We can un-deprecate it. It has come up before when I discussed open enums with @kupiakos.

I'd rather first make it a hard error if we are attaching new rules to it.

The C standard says that such an enum is compatible with $int. However, CFI implementations violate the C standard and do not allow function signatures to type-pun between enums and their underlying $int. IMO that's quite rude of them, standards exist for a reason

Convince clang that integer normalization should also normalize enums to their underlying integer type. (Either for all enums, or only for enums with an explicit underlying type.) We already require integer normalization for CFI to work with Rust, and it seems reasonable to include enums in that, at least those with an explicitly declared underlying integer type.

One thing I would say about this option is that having Rust enums correspond to integers is not really perfect compatibility with the C standard either. The C standard has a non-transitive ABI-compatibility notion, and different enums are not compatible with each other, so having Rust enums correspond to integers means that Rust would have no way of declaring a type whose ABI is exactly the same as a C enum.

Now, being compatible with integers is probably good enough, but it means that Rust will permit some function calls that are not allowed in C code by the standard, even if they are basically forced to work in practice.

Agreed. Accepting more code than the standard is fine. Accepting less is not.

Declaring that Rust always allows such cases is still risky. That's what we saw happen for pointer types. We declared that we accept pointee type mismatches, and now we are in trouble because a sanitizer showed up and rejects these cases, which is compliant with the C standard.

For example. The userspace version of CFI allows a function be list multiple signatures (to handle subclassing), which means that

void foo(enum bar my_enum);

could have been represented as having two allowed signatures: fn(u8) and fn(my_enum). In that case, CFI would trip if you cast the function pointer to fn(my_other_enum) and invoked it. Rejecting this case is compliant with the C standard.

Now, userspace CFI is not implemented like this (and kernel CFI can't be), but any time we accept such a mismatch we should keep this risk in mind.

On the Rust side, we can always use the maximally permissive signature (in this case, a bare integer). A function with signature void foo(int) is legal to call as if it were void foo(enum my_int_enum), according to the standard. So there is no danger here, if we represent extern "C" fn(MyIntEnum) as accepting a c_int for CFI purposes.

I think that is fine. Rust functions will then always have both signatures. That satisfies both the Rust-to-Rust and FFI guarantees we would make.

Miri currently makes this UB and we haven't gotten reports about this, so making them fully ABI-compatible does not seem to be in high demand.

I guess it is common in codes with ffi where they define int in one side and C enum in the other side. You can't use Rust enums in some C patterns (e.g. when enums are used as bitfields). Miri does not support ffi, and that's why there weren't any report. (My miri supports ffi, and that's how I noticed this problem)

Whatever approach we take, we need to ensure that Rust can accept a C enum as something other than a Rust enum because of C enums' semantics accepting underlying values that aren't named variants. I think that locks us into requiring some version of (1).

Navigation Menu

Repository navigation

What should the ABI compatibility rules be for `repr($int)` enums? #628

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

What should the ABI compatibility rules be for repr($int) enums? #628

Description

Activity

Jules-Bertholet commented on Oct 29, 2026

Darksonn commented on Oct 29, 2026

Darksonn commented on Oct 29, 2026

RalfJung commented on Oct 29, 2026

Darksonn commented on Oct 29, 2026

RalfJung commented on Oct 29, 2026

Darksonn commented on Oct 29, 2026

RalfJung commented on Oct 29, 2026

Darksonn commented on Oct 29, 2026

Jules-Bertholet commented on Oct 29, 2026

RalfJung commented on Oct 29, 2026

hkalbasi commented on Oct 30, 2026

CAD97 commented on Nov 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

What should the ABI compatibility rules be for `repr($int)` enums? #628