i have a small doubt concerning the behavior of cache in a buffer overflow attack.
consider the following sequence of events:
1. suppose stack is in d-cache.
2. assume no context switch occurs.
3. return statement causes a return to a location on the stack and %eip points to stack address.
4. processor looks for stack address(code) in i-cache.
5. L1 cache miss. bring old stack from L2.
6. so new stack has not been written back to memory and old stack is loaded into i-cache.
7. stack coexisting in both d and i cache.
only one of the changes is actually written back.
moreover since stack in i-cache loaded from L2 cache.
If such a sequence of events occurs then the buffer overflow attack will be foiled possibly with an invalid machine instruction exception.
is this sequence of events possible?
i understand this can now be handled by removing execute permissions from stack code but otherwise how was this handled in the pentium based systems.