Address Windowing Extensions ( AWE ) technology on Windows platforms with Intel CPUs

Address Windowing Extensions ( AWE ) technology on Windows platforms with Intel CPUs

Hi everybody,

It is a well known problem ( or limitation ) that a 32-bit application on some 32-bit Windows platforms can not allocate more than 2GB of memory.

However, some 32-bit Windows platforms, for example Windows XP Professional or Windows Server 2003, support Address Windowing Extensions ( AWE ) technology ( designed by Microsoft ) and a 32-bit application could allocate more than 2GB of memory.

Please take into account that Intel CPUs should have support for a Physical Address Extensions ( PAE ) feature.

Enclosed is a very simple test-case in 'AweTestAppV1.zip' ( Visual Studio 2005 and up ) that tries to allocate different amounts of memory and allows to verify if a 32-bit application could allocate more than 2GB of memory on Windows platforms that support AWE.

Notes:
- An Intel CPU PAE feature is supported even on Intel Pentium(R) 4 CPU
- A 32-bit Windows platform must support AWE ( for example, Windows XP Home doesn't support AWE )
- Please take a look at a small Readme.txt file for some technical details
- For more information about AWE technology visit: http://msdn.microsoft.com/en-us/library/windows/hardware/gg487508.aspx

Best regards,
Sergey

AnhangGröße
Herunterladen awetestappv1.zip5.31 KB
47 Beiträge / 0 neu
Letzter Beitrag
Nähere Informationen zur Compiler-Optimierung finden Sie in unserem Optimierungshinweis.
Bild des Benutzers iliyapolak

Thanks Sergey!
I have already downloaded your source code and extensive testing will be performed later.

>>...I have already downloaded your source code and extensive testing will be performed later...

Excellent and thank you. Here are two additional notes:

- Start with 1.5GB ( you will see a set of defines / by default I set it to 2.5GB )
- 1.86GB was the maximum amount of memory the application was able to allocate in the Release configuration on a 32-bit Windows platform ( when LARGEADDRESSAWARE is enabled )
- Use the Windows Task Manager in order to see how memory is allocated
- When defines:

#define _INIT_ALLOCATED_MEMORY_PASS1
#define _INIT_ALLOCATED_MEMORY_PASS2

are uncommented it takes some time to complete the initialization of memory. I recommend to comment out both at the beginning of tests:

// #define _INIT_ALLOCATED_MEMORY_PASS1
// #define _INIT_ALLOCATED_MEMORY_PASS2

Bild des Benutzers iliyapolak

>>>Excellent and thank you. Here are two additional notes:>>>
Tomorrow I will post my results.

Btw AWE memory mapping is more secure because it won't be paged out.

>>Tomorrow I will post my results...

Excellent. Will you be able to make a couple of screenshots?

>>...Btw AWE memory mapping is more secure because it won't be paged out...

I'm simply interested...

Where did you find that information? Is it a statement from Microsoft?

Bild des Benutzers iliyapolak

>>>Excellent. Will you be able to make a couple of screenshots?>>>
I could , but I do not know how to make a screenshots(I'm not joking).

>>>I'm simply interested...

Where did you find that information? Is it a statement from Microsoft?>>>

I strongly recommend you to read the ultimative source of Win OS knowledge a book written by Mark Russinovich titled "Windows Internals" in its 5 and 6 edition.
Sadly there you won't find any source code examples, but the amount of information transmitted to the readers is amazing and very insightful.
http://www.amazon.com/Windows-Internals-Part-Covering-Server/dp/07356487...

>>...I could , but I do not know how to make a screenshots...

There is a 'Print Screen' button at the top of keyboard ( usually in a right part ). After you pressed it create a new image in MS Paint and press 'Ctrl+V' ( Paste ), edit if something needs to be corrected or marked, and press 'Save As' to save the image as a jpg-image. Jpg-format is a significantly smaller in size then default bmp-format.

Bild des Benutzers iliyapolak

>>>There is a 'Print Screen' button at the top of keyboard ( usually in a right part ). After you pressed it create a new image in MS Paint and press 'Ctrl+V' ( Paste ), edit if something needs to be corrected or marked, and press 'Save As' to save the image as a jpg-image. Jpg-format is a significantly smaller in size then default bmp-format.>>>
Thanks
Today I was busy with my project(I'm writing GUI calculator in Java which will be able to compute special and elementary functions).Tomorrow I will test your application.

Hi Iliya,
I'll be ready for any questions related to these tests.
Best regards,
Sergey

Bild des Benutzers iliyapolak

>>>I'll be ready for any questions related to these tests.>>>
Today I will do it.
Will you be interested in my GUI calculator testing?

>>...Will you be interested in my GUI calculator testing?

Iliya,
I'm very sorry but UI is no longer my specialization. However, if you make some screenshots of the application I'll be able to provide some limited feedback. Is it OK?
Best regards,
Sergey

Bild des Benutzers iliyapolak

Thanks .Later I will send a fully working app.

Now regarding your project I'm struggling with "bcdedit.exe /set PAE ForceEnable" option and it seems not effective I have memory failure allocation reportred by your app.
I'm using win 7 and edited boot option with bcdedit.exe.

Bild des Benutzers iliyapolak

Tried also "bcdedit.exe set/ increaseuserva 3072" option wchich also was not effective(else statement is executed every time).
Please post your OS configuration.What options were set in order to successfuly testr your app?

>>...I'm using win 7 and edited boot option with bcdedit.exe...

I understood that you have Windows Server 2003 and I don't understand why are you using Windows 7.

So, Please take a look at a Readme.txt file enclosed with the project. There is a link to a Microsoft's article about AWE and I also made a couple of notes. The most important are as follows:
...
1. ...test Microsoft AWE technology on Windows Server 2003

2. /3GB switch in Boot.ini file has to be used

3. Initial size of the Virtual Memory file has to be equal or greater than 3GB
...
In order to set an Initial size of the Virtual Memory file please follow these steps:

On Windows Server 2003

[ Start button ] -> Settings -> Control Panel -> System -> Advanced -> Performance Settings -> Advanced -> Change Virtual Memory -> Select 'Custom Size' radio button -> Set 'Initial size (MB)'=3072 and 'Maximum size (MB)'=8192 -> Press 'Set' button -> It is possible that a request to reboot the computer will be asked.

Best regards,
Sergey

Bild des Benutzers Patrick Fay (Intel)

Hello Sergey,
Also note that 32bit apps on 32bit windows can use up to 3GB of memory.
There is an article on this at http://msdn.microsoft.com/en-us/library/aa366778.aspx
Pat

Bild des Benutzers iliyapolak

>>>I understood that you have Windows Server 2003 and I don't understand why are you using Windows 7.>>>
No.I 'm using Win 7,but I have a few VM's with various OS's already pre-installed.
Today I will create another VM with Win Server 2003 and extensive testing will be performed.
I will report about the results.

>>>>On Windows Server 2003

[ Start button ] -> Settings -> Control Panel -> System -> Advanced -> Performance Settings -> Advanced -> Change Virtual Memory -> Select 'Custom Size' radio button -> Set 'Initial size (MB)'=3072 and 'Maximum size (MB)'=8192 -> Press 'Set' button -> It is possible that a request to reboot the computer will be asked.>>>

Thanks.

Hi Iliya,

>>No.I 'm using Win 7,but I have a few VM's with various OS's already pre-installed.

Do you use a VirtualBox software?

>>Today I will create another VM with Win Server 2003 and extensive testing will be performed.

I hope that it won't take too much your time. The test is really simple and I expected that it could be completed in a couple of minutes.

Thanks for all your efforts.

Bild des Benutzers iliyapolak

>>>Do you use a VirtualBox software?>>>
No VMware workstation 7.

>>>>Do you use a VirtualBox software?
>>
>>No VMware workstation 7.

Thanks.

>>...I hope that it won't take too much your time. The test is really simple and I expected that it could be completed in a couple of minutes.

Iliya, please don't spend too much time on it unless you're really interested in AWE technology.

Hi Patrick,

>>...There is an article on this at http://msdn.microsoft.com/en-us/library/aa366778.aspx...

Thanks for the link and it has another link to a very good article:

What is 4GT?
http://technet.microsoft.com/en-us/library/cc786709(WS.10).aspx

Take into account that /3GB switch in the Boot.ini file could be used on the following systems:

- Windows Server 2003 x64 Editions family
- Windows XP Professional Edition
- Windows 2000 Datacenter Server
- Windows 2000 Advanced Server
- Windows NT Server 4.0 Enterprise Edition

>>...
>>What is 4GT?
>>http://technet.microsoft.com/en-us/library/cc786709(WS.10).aspx

I also recommend to look at these articles:

How 4GT Works
http://technet.microsoft.com/en-us/library/cc784475(v=ws.10).aspx

4GT Technical Reference
http://technet.microsoft.com/en-us/library/cc778496(v=ws.10).aspx

4GT Tools and Settings
http://technet.microsoft.com/en-us/library/cc737977(v=ws.10).aspx

PAE x86 Technical Reference
http://technet.microsoft.com/en-us/library/cc728455(v=ws.10).aspx

Bild des Benutzers iliyapolak

>>>Iliya, please don't spend too much time on it unless you're really interested in AWE technology.>>>
Why not:)
I'm interesting in system programming.Sometimes I spent quite a lot of time with the kernel debugger while I'm digging into Windows internals.
It is very interesting when you can see various critical data structures like KDPC(kernel deferred procedure calls) or PCR(processor control region) beign parsed and exposed by debugger.Now i'm working on kernel module which will be able to perform various manipulations on those structures.

Bild des Benutzers iliyapolak

>>>I also recommend to look at these articles:>>>
Thanks
I would also recommend you to read "Windows Internals" books.

>>Today I will create another VM with Win Server 2003 and extensive testing will be performed.

What is your progress with installing Windows Server 2003 in the VM?

Bild des Benutzers iliyapolak

Has been already installed,soon I will post the results.

Hi Iliya,

>>...Has been already installed, soon I will post the results...

Just to clarify the task: At least 2.5GB of memory has to be allocated for the 32-bit test application ( source codes are provided ) on Windows Server 2003 ( 32-bit ). Let us know as soon as the test is completed.

Best regards,
Sergey

Bild des Benutzers iliyapolak

>>>Just to clarify the task: At least 2.5GB of memory has to be allocated for the 32-bit test application ( source codes are provided ) on Windows Server 2003 ( 32-bit ). Let us know as soon as the test is completed.>>>
Thanks.
Soon I will post the results.(I'm quite busy with my projects and learning:))

If you are interested I'm posting an update to my "intrinsincs" project.Please look at my sine functions thread.

Bild des Benutzers iliyapolak

@Sergey
I'm switching to Win XP Pro because I have a stability issues with the Win Server 2003.
Today I will run your code.
Sorry for beign late with the tests.I was quite busy last few days.
Btw.I created a new thread in "AVX and CPU instructions forum" where I described a strange issue related to inline assembly code.
If you are interested please look at the code.
Thanks in advance.

Hi Iliya,

Thanks for the update.

>>I'm switching to Win XP Pro because I have a stability issues with the Win Server 2003.

Do you mean when it is installed in VMware Workstation 7?

>>Btw.I created a new thread in "AVX and CPU instructions forum" where I described a strange issue related to inline assembly code.
>>If you are interested please look at the code.

I saw and I'll take a look at it during the weekend.

Bild des Benutzers iliyapolak

>>>Do you mean when it is installed in VMware Workstation 7?>>>
Yes I have 5 VMs 4 of them are Win OSs(3 Win Server 2008 and 1 Win Server 2003) and one is Linux hacking distro Backtrack 4.
Now those VMs are used to test downloaded malware infections with the help of static and dynamic analysis.
Regarding Win Server 2003 stability issue it probabaly stems from my kernel debugging experiments where I usually try to perform modification of crucial data structures.

>>>I saw and I'll take a look at it during the weekend.>>>
Thank you.

Here is update... I have some progress and finally I'm able to see how it works. However, it will take some time before a final decision about applicability of AWE-based solution will be made. There are lots of concerns and the most important is AWE is Not a portable solution.

Anlagen: 

AnhangGröße
Herunterladen awetestapp3.jpg74.94 KB
Bild des Benutzers iliyapolak

>>>Here is update... I have some progress and finally I'm able to see how it works. However, it will take some time before a final decision about applicability of AWE-based solution will be made. There are lots of concerns and the most important is AWE is Not a portable solution.>>>

I can see that you were able to allocate 100% of the requested pages.Was your previous AWE mem allocation tests so successful?

>>...Was your previous AWE mem allocation tests so successful?

No because only yesterday I finally created a right environment for AWE tests. I'm still verifying results and sources because there are many questions. I will also need to spend more time on documentation.

By the way, a long vertical red line is not an allocation of memory pages. It is some kind of moving cursor.

Bild des Benutzers iliyapolak

>>>No because only yesterday I finally created a right environment for AWE tests. I'm still verifying results and sources because there are many questions. I will also need to spend more time on documentation.>>>

Can you send me for testing your source code?
I need to download 32-bit Win 2003 OS and I will test your AWE code.

>>...Can you send me for testing your source code?

Yes, of course. I've created two more little projects ( there are 3 in total now ) but the last one is in a very raw-state. I will upload all updated projects in about one week or later. I also would like to mention that I really underestimated efforts related to AWE tests. It took more than a couple of days to create a right environment in a VMware Player 5.0.1 ( build-894247 ).

In overall, I clearly see that AWE is a more advanced version of a 20+-year-old Expanded Memory Specification ( EMS ) technology for 16-bit operating systems. In the past, Microsoft and Intel had problems with address space limitations twice:

- it wasn't enough for applications to have a 20-bit address space in a 16-bit MS-DOS & Windows 2.x/3.x ( without Win32s extension )
- it wasn't enough for applications to have a 32-bit address space in a 32-bit Windows

Bild des Benutzers iliyapolak

>>>In overall, I clearly see that AWE is a more advanced version of a 20+-year-old Expanded Memory Specification ( EMS ) technology for 16-bit operating systems. In the past, Microsoft and Intel had problems with address space limitations twice:>>>

It should be more advanced partly because of more advanced and capable hardware.

>>>Yes, of course. I've created two more little projects ( there are 3 in total now ) but the last one is in a very raw-state. I will upload all updated projects in about one week or later. I also would like to mention that I really underestimated efforts related to AWE tests. It took more than a couple of days to create a right environment in a VMware Player 5.0.1 ( build-894247 ).>>>

Thank you.
Regarding VMware player why do not you use VWware workstation?

>>...why do not use VWware workstation?

It is up to you what VMware version should be used on your computer(s). I don't want to start a discussion about advantages or disadvantages of different versions of VMware.

Bild des Benutzers iliyapolak

Zitat:

Sergey Kostrov schrieb:

>>...why do not use VWware workstation?

It is up to you what VMware version should be used on your computer(s). I don't want to start a discussion about advantages or disadvantages of different versions of VMware.

I also do not want to talk about the advantages and disadvantages of the variuos versions of VMware software.For my needs Workstation version is must for  me.

Bild des Benutzers iliyapolak

Zitat:

iliyapolak schrieb:

Quote:

Sergey Kostrovwrote:

>>...why do not use VWware workstation?

It is up to you what VMware version should be used on your computer(s). I don't want to start a discussion about advantages or disadvantages of different versions of VMware.

I also do not want to talk about the advantages and disadvantages of the variuos versions of VMware software.For my needs Workstation version is must for  me.

My future plan includes usage of 10-12 virtualized 0S's simlating network segment composed from the various varsion of patched and unpatched Windows OS's I would like to test malware propagation and infection.I have already downloaded 3 DVD filled with various malware.

>>My future plan includes usage of 10-12 virtualized 0S's simlating network segment composed from the various varsion of
>>patched and unpatched Windows OS's I would like to test malware propagation and infection. I have
>>already downloaded 3 DVD filled with various malware.

Iliya,

I'm very sorry but the subject of the thread is Address Windowing Extensions ( AWE ) technology on Windows platforms with Intel CPUs and I really don't understand why you're continuing discussions on absolutely unrelated subjects like malware propagation and infection.

You could create a new thread in http://software.intel.com/en-us/forums/watercooler-catchall for any subjects which do not violate Intel's Forums Policies.

Once again, PLEASE do not "pollute" threads with unrelated subjects and I know that sometimes it is not an easy thing because we're on a forum.

Thanks in advance.

Best regards,
Sergey

Bild des Benutzers iliyapolak

It is not easy to stay focused on main thread topic.

Sorry for posting quite often off topic or unrelated information.

Bild des Benutzers iliyapolak

If you are interested I have found a great tool written by guru M.Russinovich the tool is named RAMMap and is able to produce very detailed memory usage.

There is also option to display data related to AWE technology.

http://technet.microsoft.com/en-us/sysinternals/ff700229.aspx

>>There is also option to display data related to AWE technology.
>>
>>http://technet.microsoft.com/en-us/sysinternals/ff700229.aspx

Thanks and I'll take a look.

Bild des Benutzers iliyapolak

>>>Thanks and I'll take a look.>>>

If you are not already using it, I would like to advise you to download whole Sysinternals toolkit.You can find there plenty of outstanding tools and very helpful.

The screenshoot I've attached demostrates memory layout of a 32-bit Windows NT operating system:

On some 32-bit WIndows operating systems ( which are almost at the end of support by Microsoft ) AWE changes that layout and more than 2GB of memory could be allocated ( out of 4GB ). However, since AWE can not be considered as a universal API that solution requires a significant amount of code modifications in an existing software system ( a port to a 64-bit Windows is significantly easier! ).

Let me know if somebody needs C/C++ projects I've created to research AWE capabilities.

Anlagen: 

Melden Sie sich an, um einen Kommentar zu hinterlassen.