Visual Studio Code – Ensuring x64 compliance of custom ASM-function in clang-cl

Making calls from inline asm is generally not well supported. Avoid whenever possible.
The compiler only scans the inline asm block to see what registers are potentially clobbered; it doesn’t assume that call instructions in asm are to functions that follow the standard calling convention for this target (otherwise why would you be using inline asm in the first place?) So it’s a huge pain to do it safely, same for x86-64 System V (Calling printf in extended inline ASM – using GNU C inline asm you also have to declare all the register clobbers yourself, as well as take care of the red-zone since there’s no way to declare a clobber on that.)

Your idea of using inline asm to leave values in regs and block tail-call optimization is a good idea. But the implementation in your self-answer with two separate asm() statements doesn’t do anything to stop the compiler from stepping on RBX with the instructions it emits for code outside the asm statements. A different compiler or version could easily break your code by picking RBX as a temporary instead of RAX when compiling that code between the asm statements. (And since you didn’t use __attribute__((noinline)), code from parent functions could be scheduled here.)

You can write it in a way that discourages the compiler from stepping on your registers. Make those values needed in those registers after the call (as inputs to an empty asm statement), so the asm you want is the only efficient choice. That makes it a lot less likely that this will break in practice.

class ExecutionStateJIT;
using Func = void ();

void interruptEntry_safer(ExecutionStateJIT& state, Func pAddress)
{
    register void *state_addr asm("rbx") = &state;
    // strict-aliasing violation, see alternate version that's safe without -fno-strict-aliasing
    register auto* pTemp asm("rdi") = * *((void***)&state);  // two derefs
    // register ... asm("rbx") is actually redundant since I also used specific-register constraints in the asm statement

    // request those vars in RDI and RBX respectively
    asm volatile("" :: "D"(pTemp), "b"(state_addr));  // make sure they're actually loaded before
    pAddress();  // this just uses a function-pointer arg that was already in a register, doesn't need to touch any others

    // prevent a tailcall which would restore RDI and RBX before calling
    asm volatile("" :: "D"(pTemp), "b"(state_addr));  // and still wanted in these call-preserved registers after
}

Using register T foo asm("regname") local register variables lets you ask for values in any of R8-R15 which don’t have specific-register constraint letters, forcing an "r"(var) constraint to pick a specific register. (And for many other ISAs, there aren’t letters for any single registers.) It’s not actually needed here because the "D" and "b" constraints require RDI and RBX respectively.

Godbolt shows it works in GCC and clang (-masm=ms for GCC, and -target x86_64-w64-windows-gnu for Clang. As a bonus, this doesn’t require very-recent Clang for -masm=intel to apply to asm statements, since the actual templates are empty. The action is in the constraints, requiring the compiler to have both values in the registers we want, but without any

Your code also violates the strict-aliasing rule by pointing a void ** at an object of a different type. Only [unsigned] char* and pointers to objects declared with __attribute__((may_alias)) can be pointed at arbitrary things in GNU C. But for compat with MSVC, clang-cl probably enables -fno-strict-aliasing.

class ExecutionStateJIT;
using Func = void ();
void interruptEntry_safer_strict_aliasing(ExecutionStateJIT& state, Func pAddress)
{
    using voidp = void*;
    using aliasing_voidp = __attribute__((may_alias)) voidp;
    // aliasing_voidp is a pointer-to-void (e.g. 8 bytes on x86-64).
    // aliasing_voidp*  can be pointed at any object safely, to let us load a void*
    void *state_addr = &state;
    auto* pTemp = *(aliasing_voidp*)state_addr;  // like memcpy but alignment guaranteed because no __attribute__((aligned(1)))
    pTemp = *(aliasing_voidp*)pTemp;

    asm volatile("" :: "D"(pTemp), "b"(state_addr));  // make sure they're actually loaded before
    pAddress();  // this just uses a function-pointer arg that was already in a register, doesn't need to touch any others
    // prevent a tailcall which would restore RDI and RBX before calling
    asm volatile("" :: "D"(pTemp), "b"(state_addr));  // and still wanted in these call-preserved registers after
}

Both of these compile to the same asm as yours with current versions of GCC and clang, and the source is shorter and easier to read (if you know GNU C inline asm). The point is that they will more reliably do so with future versions and even if inlined into other surrounding code.

interruptEntry_safer_strict_aliasing(ExecutionStateJIT&, void (*)()):
        push    rdi
        push    rbx
        mov     rbx, rcx               # state_addr
        sub     rsp, 40
        mov     rax, QWORD PTR [rcx]   # first pTemp
        mov     rdi, QWORD PTR [rax]   # second value of pTemp
        call    rdx
# clang puts a NOP here for some reason
        add     rsp, 40
        pop     rbx
        pop     rdi
        ret

I didn’t use __attribute__((noinline)) on my versions since even in a use-case where they do inline into a caller (e.g. -flto link-time optimization), the asm statements hopefully convinces the compiler not to do something else with RBX or RDI in that window between the asm statement and the call, if it is moving code around to try to schedule it more efficiently.