I don't like x86 instruction set because it is full of exceptions and arbitrary historical conventions. On top of this every new group of instructions follows the suit. It's not CISC anymore, more like OCISC standing for 'Over-Complicated Instruction Set Computer'.
I'm writing a quite nice x86 assembler in python to get a backend for compiler I'm working on. The trouble is that it is probably impossible for me to use anything except the smallest subset of x86 if I want to keep it simple and easy to read. This far I think I've done somewhat well as long as I carefully read the instruction set reference to avoid all the possible oddities I might locate.
Partially I'm frustrated because x86 has forced me through thick swamp to actually be able of doing anything like this. It's not completely bad thing since I've learned to treat superfluous complexity through avoidance. Mostly it is an annoyance and is able of complicating whatever code generator library one needs to implement for the platform.
Could hardware designers at intel provide an alternative aside x86-instruction set? Requirements would be that writing an assembler/code generator for it must be simple, and it must reach performance greater or alike to the x86 machine code. It could be also possible to achieve this by just picking up a collection of instructions that are easy to deal with.
As I've attempted to write an assembler, I've figured there's couple of changes into instruction set that could make it considerably easier to implement code generators. Fixed instruction length is one of them. Simple thing as encoding instruction length&arguments into varying-width opcode and getting rid of modrm/sib/address size -bytes would already do it lot easier for writing instruction encoders.
There's also one very sneaky thing you could do, ok. Here's how it'd go: Write a manual that describes 50-100 instruction forms that, when implemented would allow writing as efficient programs as before. Promote compilers, assemblers and software that are using this new suspended instruction set. Then, one pretty day, release a new architecture that contains this subset and completely new instruction set, but contains software emulation layer for the whole messy history of the x86.
The idea in this approach lies behind the fact that the computing power increases exponentially every three years. Introduce and force the restricted instruction set for couple of years for the newly written software. In this time the current software stop being latest edge in hogging computing performance and emulating them will provide sufficient performance for anyone still using them.