I have an RTOS 64bit which uses SYSCALL/SYSRET for system calls. They work fine if this OS is executed "native" on an Intel machine.
Now if I use a VMM (under development, 64bit) and start the OS as 64bit guest VM the SYSCALL causes an invalid opcode exception. The EFER.SCE bit is set and no LOCK prefix is used. I've also tried to use SYSENTER/SYSEXIT in this OS and they work just fine.
I'm not sure if I have setup the VMCS correctly. Are there any hints or special settings in VMCS regarding SYSCALL/SYSRET when running virtualized?
Intel machine has an Intel Xeon CPU W3520 with VT-x/VT-d enabled.
Any help is appreciated, thx in advance!