What are the rules for AVX/YMM registers under the Windows 64-bit ABI?
I dont think there is any change around ABI. As XMM registers are lower 128-bit part of YMM registers. In the end they are same registers, however extended to 256bits. So, in case of function calls the lower parts will be used to pass floating point arguements.
The story of the week is back to a wait of about a year from the introduction of YMM register machines until a SP for Windows 7 and server will support them.
Except that it is more than just calling conventions, the ABI also covers exception handling, and it also includes rules for prolog/epilog of functions which needs to followed for it to work properly. That is what I am particularly concerned about.
The bottom half of the YMM registers follow the x64 ABI rules for XMM registers. The top half is considered volatile.