Using SYSCALL/SYSRET in a 64bit guest OS (running as a VM) causes #UD

Using SYSCALL/SYSRET in a 64bit guest OS (running as a VM) causes #UD


I have an RTOS 64bit which uses SYSCALL/SYSRET for system calls. They work fine if this OS is executed "native" on an Intel machine.

Now if I use a VMM (under development, 64bit) and start the OS as 64bit guest VM the SYSCALL causes an invalid opcode exception. The EFER.SCE bit is set and no LOCK prefix is used. I've also tried to use SYSENTER/SYSEXIT in this OS and they work just fine.

I'm not sure if I have setup the VMCS correctly. Are there any hints or special settings in VMCS regarding SYSCALL/SYSRET when running virtualized?

Intel machine has an Intel Xeon CPU W3520 with VT-x/VT-d enabled.

Any help is appreciated, thx in advance!

5 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

Hello and thanks for your post. Such kind of instructions (including SYSENTER/SYSEXIT && SWAPGS) use MSRs to perform operations. Have you saved the state of thos MSRs?irp

Thanks for your answer,

interruptrequestpacket. What MSRs do you mean shall be saved?

Do you mean I should save them on a VM-Transition? The point is, a VM-Transition is unwanted when a SYSCALL/SYSRET is used. There is also none (usually) when using SYSENTER/SYSEXIT.

Thanks in advance!



Hello!Well, we can take SYSCALL as an example.After the processor saves the RFLAGS and the next instruction after the call, it has to get the ring0 privileged code to be executed, ie the target instruction and stack ptr.
To perform this operation guest OS has to ensure that the segment of the instruction (base, access rights, limit) loaded by the call (processor will reads a nonNULL selector from IA32_STAR to get the target code selector) correspond to the descriptor in the GDT. Also to use this instructions OS need to check if CPUID.0x80000001.EDX[bit11] = 1 (if you don't know how ask for it!).So maybe you can have some exception that causes a VMexit? This is what I think, please explain me if you have another idea.But if you choose to implement SYSCALL/SYSRET with this preview, you will need to save IA32_STAR (told you before why, also for the stack segment for the stack ptr), IA32_LSTAR (used by the processor to get the target instruction) , IA32_FMASK (something about flags); then load the host state, and finally restore the guest with VMentry operation.Regards,irp

Hello and thank you very much for your input.

What you have mentioned is the situation if both, VMM and VM use SYSCALL and therefore need to save the mentioned MSRs during a VM-Transition.

My situation is a bit different. VMM does not use SYSCALLs and therefore these MSRs don't have to be stored/restored during a VM-Transistion as they remain untouched.

Anyway, meanwhile the problem has been solved. The problem was the SCE bit in EFER MSR. This bit has been set in the VMCS of the VM but has not been set in VMM itself. Setting it in VMM allows the use of SYSCALL. (There has been similar problem with NXE bit.)

Thanks a lot for answers. This thread might be closed.

Leave a Comment

Please sign in to add a comment. Not a member? Join today