| Last Modified On : | October 17, 2008 4:05 PM PDT |
Rate |
|
By Dino Esposito
Unless you've spent the past couple of years buried in an inaccessible and dark cave with no connectivity, you probably know that a .NET application is made of managed code. But what's managed code exactly? According to the Microsoft official definition, managed code is a standard Windows* executable file that requires a special manager module to run. This module is the Common Language Runtime* (CLR). The atomic unit of processing that the CLR understands and recognizes is the assembly. An assembly is a group of one or more managed modules optionally including resource files. Not only can assemblies include multiple modules, but they can also span over multiple files.
Managed modules are produced by specialized .NET compilers that translate high-level languages (C#*, Visual Basic .NET*, Managed C++*, J#*) to the Intermediate Language* (IL). IL is the native language of the CLR—a sort of x86 assembly for such a virtual CPU. (Do not confuse the .NET assembly, which is a compiled module, with the x86 assembly language, which is instead a low-level programming language.)
In a nutshell, a .NET application is made of one or more assemblies that require an instance of the CLR to execute. In this article, I'll first briefly review the architecture of the CLR attempting a basic comparison with the Java* Virtual Machine (JVM). Next, I'll analyze the internal layout of the managed code and explain how to create assemblies. Finally, I'll discuss the code execution process in .NET and the language neutrality feature that makes it possible for you to write code in C# and reuse it from Visual Basic .NET applications.
Conceptually, the CLR looks a lot like the JVM. The CLR is a totally controlled run-time environment that manages the execution of intermediate code. The CLR intermediate code looks like Java's bytecode. In addition, but again like the JVM, the CLR provides applications with built-in services such as garbage collection, debugging and profiling facilities, threading, exception handling, versioning, and more. (The JVM does not provide the same set of services as the CLR, but the resemblance between the two models is apparent.)
The similarity between the JVM and the CLR, though, is only conceptual and ceases as soon as you dig deeper into the virtual machines. Architecturally speaking, in fact, a number of significant features do exist and contribute to keep the JVM and the CLR on radically different planes. The most important difference you can notice between the virtual machines is about their own way of processing the code.
The JVM still works like an interpreter and the just-in-time (JIT) compiler is a mere optimization. It runs a loop in which any line of bytecode is read and then compiled and executed. At the end of the instruction, the control passes back to the internal interpreter that continues reading and compiling the next lines until the end of the application is reached.
The CLR employs a significantly different model. The CLR has no silent interpreter module to rule the execution process. A .NET compiler, in fact, generates an intermediate code in which any call to any method references a standar d stub program. The IL is basically the language of a stack-based machine. This is a first huge difference because in this way the intermediate code executes sequentially and automatically as if it were compiled to machine code. The stub calls a standard program that just-in-time compiles ("JIT'd") the IL code to native code the first time called. After that, the reference to the stub code is replaced with the actual address of the JIT'd code throughout the whole program. In this way, the execution process is more effective, faster, and in no way different from running native machine code.
The .NET compiler is a tool capable of generating IL code starting from a source written in a number of different languages. Aside from the most popular languages like C# and Visual Basic .NET, a fair number of less famous (but more fancy) languages are supported. The list includes COBOL*, SmallTalk*, Perl*, Eiffel*, but also APL*, ML*, and Oberon*. It's amazing to recall that one of the very first examples of managed applications that Microsoft demonstrated at PDC 2000 was written exclusively in COBOL .NET.
The output of a .NET compiler is a managed module that contains IL code as well as metadata tables. More in detail, the actual content of a managed code includes:
- PE header
- CLR header
- Metadata
- IL code
The PE header is the standard Win32* portable executable (PE) header. Any Windows executable must have a PE-compliant header in order to run on a Win32 platform. Obvious exceptions are all the Win16* programs that the loader recognizes as such and handles differently. The .NET platform was introduced in the summer of 2000 and released February 2002. In all this time, Microsoft released only one new operating system (OS)—Windows XP. Nevertheless, .NET executables can run on existing Win32 operating systems like Windows 2000, Windows NT 4.0 and even Windows Millennium Edition and Windows 98. Where is the magic? Any .NET executable looks like a plain old Win32 application and this is the key factor that makes it run on Win32 systems. I'll have more to say on this point later on.
Aside from the initial old-style PE header, a .NET executable also includes a CLR header where are stored information like the version of the CLR required, the entry point in the code, any resources, size, and offset to the metadata.
The metadata are the .NET counterpart of a COM-type library. Metadata tables describe in detail the classes defined and referenced in the source code. Metadata also include a description of the assembly, all the necessary identity information (name, version, culture, public key), the security permissions requested to run, and the other assemblies dependency list. In addition, for every type of information you will find the name, the visibility (public, protected, private), the base class, and the interfaces implemented. Every method, field, property, event, and nested type is listed along with its attributes. When managed code is executed, the CLR loads metadata into memory and utilizes it to discover information about your classes.
The behavior of the managed module is determined by the IL code—the output that any .NET compiler generates after processing the source code. The IL code is not machine code and, as such, it is not directly executable. It will be compiled on the fly to CPU-specific code and executed. The code remains in memory as long as the application is up and running. The compiled code is not written to disk but is held in memory using a sort of memory mapped file. When the application exits, any block of its JIT'd code is just discarded.
The .NET Framework comes with a utility called ngen.exe that you can use to pre-compile managed executables so that they ship already in the final x86-compliant form. The ngen syntax is:
ngen.exe [assembly]
The file is created in an internal folder and shows up in the global assembly cache (GAC) under c:winntassembly. (More later.) Unfortunately, though, the executable created by ngen.exe cannot run without the original IL code. In addition to this, the current version of ngen.exe comes with several other drawbacks insightfully described in a CodeGuru.com article* titled "JIT Compilation and Performance—To NGen or Not to NGen?" As a result, using the current version of ngen.exe makes sense only for client applications and only under particular circumstances in which the executable required too much to load. Bear in mind, however, that the time to JIT is only a portion of the time needed to set up managed code to run.
An assembly is primarily a binary managed module that contains the code that the CLR executes. An assembly, though, is more than this and especially contains more than just executable code. For instance, an assembly cannot be executed if it does not have an associated assembly manifest. A manifest is just another set of metadata tables. A manifest in particular contains the public interface of the assembly: version information, the list of the files that form the assembly, the exported types, and any resource files. The following figure shows the layout of an assembly at the highest level of abstraction.
Figure 1. The structure of an assembly.
An assembly is created by special tools—typically, a .NET compiler like csc.exe (the C# compiler), or the al.exe (the Assembly linker). The tools are distributed through the .NET Framework SDK.
The following command line shows how to create an assembly from a C# source file.
csc.exe /t:exe /out:test.exe file.cs
The /t:exe switch instructs the compiler to create an executable .exe file named test.exe. The output name is determined by the /out switch. To create an executable library, use the /t:library option. While building an assembly, you may need to reference external assemblies in much the same way as in Win32 you need to link external import libraries. In this case, you use the /r switch followed by the name of the DLL. For example:
csc.exe /t:library /out:test.dll /r:system.xml.dll
file.cs
Notice that you must indicate the assembly name for each system assemblies needed. The name of the assembly does not always match the name of the namespace you import in your source code. For example, if your code makes use of XPath classes, you might want to use the System.Xml.XPath namespace to shorten the name of the involved classes.
using System.Xml.XPath;
This line does not automatically reference the proper assembly in the binary being generated. In order to add the assembly with XPath classes to your code, you must add an explicit reference to it. The name of the assembly does not necessarily match the name of the namespace. For example, all XML classes in the .NET Framework are partitioned into four different namespaces but the assembly is only one: system.xml.dll. To become an assembly, the source code must contain at least a class or an interface definition.
An important part of the assembly is the version information. When you create a new project with Visual Studio, it automatically generates an AssemblyInfo file for you. By simply modifying the source of that file you can control the version information exposed by the assembly. The following listing shows the content of the file for a C# project.
using System.Runtime.CompilerServices;
[assembly: AssemblyTitle("")]
[assembly: AssemblyDescription("")]
[assembly: AssemblyCompany("")]
[assembly: AssemblyProduct("")]
[assembly: AssemblyCopyright("")]
[assembly: AssemblyTrademark("")]
[assembly: AssemblyCulture("")]
[assembly: AssemblyVersion("1.0.*")]
[assembly: AssemblyDelaySign(false)]
[assembly: AssemblyKeyFile("")]
[assembly: AssemblyKeyName("")]
The first block of fields indicates rather self-explanatory data. Only the Culture field requires some more explanation. The culture information is another part of the assembly's identity along with version numbers and names. Cultures refer to country-specific settings. A culture is identified by a string formed by a couple of tags. For example, "en-US" for English US and "en-GB" for English UK. Normally, the primary tag refers to the language while the secondary tag qualifies the country that utilizes that language and subsequently the settings. Cultures serve the purpose of multi-lingual and localizable applications. Assemblies with no culture information are said to be culture neutral and contain code and general settings. Assemblies marked with a specific culture tag should not contain code but only resources. In this case, they are said to be satellite assemblies.
The last three assembly version fields refer to the strong name of the assembly—a topic that I'll cover in a moment.
Multi-File Assemblies
As mentioned previously, an assembly can also be partitioned into multiple physical files. In other words, the assembly is still seen as a single logical group of constituent components but its content is split into two or more distinct files. This is not really different from partitioning a monolithic application into several dynamic-link libraries. There is no special guideline to follow when it comes to partitioning an assembly into distinct files. However, since multi-files assemblies are downloaded on demand, it's a good idea to isolate in a separate file those modules that you plan to use less frequently. When you have a multi-file assembly only one of constituent files will be the keeper of the manifest. Alternatively, you can also create an extra assembly that contains only the manifest and no other code or data. Interestingly, Visual Studio .NET does not provide you with the ability to create multi-file assemblies. For that, you must resort to the aforementioned command line tools like al.exe or csc.exe.
Private Assemblies
An assembly can be deployed to a user's machine in a variety of ways. The simplest way is just copying all the files. Because assemblies are self-referencing and expose all the information about references and dependencies, there is no need to store anything in the system registry or in the Active Directory tables. Other techniques to deploy assemblies include cabinet (*.cab) files and MSI packages. Of course, the more complex the technique is, the more features you provide to your users. However, if your goal is simply "make it run," then you just need to copy all the assemblies that form a program.
Copying all the assemblies to the same application's directory means that the assemblies are privately deployed to the application. If this does not sound particularly exciting, consider that it means that no other application will be using that assembly. All referenced types are scoped in the assemblies in the directory. Uninstalling the application is easy, too. You just delete or empty the directory and you're done.
Based on this layout, it seems impossible that an application could reference a file outside its own directory. By using a configuration file, the author (or the administrator) can instruct the application to load an assembly from another directory. However, this directory can only be a child of the main application's directory. The idea, in fact, is that a .NET application controls only its own subtree and has no power over other parts of the file system.
The configuration file you use to redirect to other subdirectories must have a particular name and structure. The name is the same name of the application with a .config extension. If the application is named test.exe, then the corresponding configuration document must be called test.exe.config and reside in the same folder. The following listing shows the typical content of a config file.
<?xml version="1.0" ?>
<configuration>
<runtime>
<assemblyBindings xmlns="urn:schemas-microsoft-com:asm.v1">
<probing privatePath="..." />
</assemblyBindings>
</runtime>
</configuration>
xmlns:urlreplace="urn:PlatformDocument" /
The <probing> element and its privatePath attribute define the alternate paths from which an application can locate its needed assemblies. Notice that if no config file is specified, the application will automatically attempt to locate all needed assemblies from the local directory, otherwise raising a file-not-found exception. The privatePath attribute can contain only relative paths. Semicolons (;) can separate multiple paths. By default, the CLR first attempts to locate any needed assemblies in various subdirectories. If unsuccessful, it then repeats the search using a .exe extension. If the operation is still unsuccessful, an exception is thrown. Satellite resource DLLs are assumed to be in subdirectories named after the satellite's culture (such as en-US and de-CH).
Shared Assemblies
System assemblies, like system.xml.dll, can be used by multiple applications. The global availability of binary components is one of the key contributors to the phenomenon sadly known as DLL Hell in previous versions of Windows. DLL Hell is a harsh but effective description of those runtime errors that arise when applications happen to use the wrong version of a shared DLL file. Any application originally ships with the right version of each constituent file. However, if one of those files already exists on the target machine, what happens is up to the setup program. In general, either the file is not copied (potentially breaking the application being installed) or the file is overwritten thus potentially breaking any installed and so far working application.
Using shared binary files was originally a good idea aimed to save valuable memory resources and offer better performance. It was so good as an idea that it soon got abused and maybe misused.
In .NET, Microsoft attempts to deliver a hassle-free system for sharing code. The first guideline for developers, though, is highly restrictive: avoid publishing globally shared assemblies if not really necessary. If you deploy your assemblies privately, as discussed in the previous section, everything works just fine. But how can we make user-defined assemblies globally available as well?
A shared assembly is not different from a private assembly. The code is identical and so is the compile process. There is just one difference between private and global components. An assembly that pretends to be shared must be signed with a pair of public/private keys specific to the vendor. An assembly signed in this way is said to be strongly named. Only strongly named assemblies can be deployed globally. By contrast, both strongly and weakly named assemblies can be deployed privately.
If many applications are going to use the same assembly, this must be placed in a common place where the CLR can easily locate and load it. Basically, DLL Hell exists because the mechanism used to define the identity of a DLL file is particularly simple and based only on the file name. Strong names are a way to name shared file to significantly reduce the likelihood of name conflicts. A strong name is made of four pieces of information:
- File name without any exten sion
- Version number
- Culture tag
- Public key
The following string is the strong name of the System.Data.dll global assembly:
System.Data, Version=1.0.3705.0, Culture=neutral,
PublicKeyToken=b77a5c561934e089
Name and version number are no way sufficient to ensure uniqueness, and culture can help to narrow the risk a bit. The fundamental piece of information that raises the level of uniqueness is the public key. A company that wants to distribute global assemblies starts by getting a pair of public/private keys. The .NET Framework sn.exe utility generates the pairs. The following command line saves the information in the specified file.
sn.exe —k mypair.keys
To guarantee a reasonable level of security, private and public keys are very long. Private keys are not exchanged or embedded with the assembly but public keys should be. To preserve space, the assembly manifest does not contain the public version of the key, but a shortcut called the public key token. The public key is actually a 64-bit hash of the private key, which is statistically proven to be unique. To strong-name an assembly, take the *.keys file you got from the sn.exe utility and pass it to a particular field in the source code.
[assembly:AssemblyKeyFile("mypair.keys")]
The compiler processes the line by reading the private key from the specified file and signing the assembly. The public key token is then inserted in the manifest. When the CLR attempts to load such a signed assembly, it will first use the public token to obtain the full public key and then decrypt the code.
Unless two companies share their own keys, there is no way that two strongly-named assemblies coincide.
Globally accessibl e assemblies are placed in a common directory (typically, c:winntassembly) known as the Global Assembly Cache (GAC). Normally, you don't manually copy or delete files to and from this repository. To do that, you should use an ad hoc tool called gacutil.exe. The following two command lines show how an assembly can be inserted into, and removed from, the GAC:
gacutil /ir assembly.dll
gacutil /ur assembly.dll
As you can see, registering assemblies with the GAC is not logically different from associating it with the system registry. Sure, the GAC is more protected than the registry (and is not accessible through an open API), but you lose the simplicity of the xcopy-style deployment. But what do you gain from a globally shared assembly placed in the GAC? The following figure shows the system's GAC when viewed through Explorer*.
Figure 2. The Global Assembly Cache.
The view in the figure is produced by a namespace extension that hides a lot of details about the internal organization of the GAC. Basically, the GAC contains different versions of the same assembly and stores them in different subdirectories. In this way, an application that registers to use version 1.345.22 of a given assembly will always be served by that version and not by another potentially incompatible version. In this scenario, side-by-side execution of different versions of the same assembly is normal. A pleasant effect of side-by-side execution is that you can't develop new versions of the same component without worrying about backward compatibility!
This contrivance, summed up to the strong name policy, makes the likelihood of DLL conflicts nearly zero. Finally, notice that shared assemblies are also tamper-resistant because of the signing process they undergo to be given a strong name.
The .NET Framework also supports a sort of delayed signing of assemblies. Delayed signing is a technique that allows you to build assemblies using only the public key. The idea is that you develop, test, and in general prepare the assembly for deployment, without signing it. You use the private key and sign it only when ready to burn the CDs. However, to avoid problems with compilers when building and referencing the assembly, you should explicitly mark it as delayed for signing.
[assembly:AssemblyDelaySign(true)]
When you're finished with preliminary operations, to complete the assembly development you sign it manually again using the sn.exe utility.
sn —R myassembly.dll
Running the code will fix any missing information in the assembly's manifest and hash the component's code for tampering.
In the .NET Framework, the assembly is the smallest unit of reuse and versioning but also the smallest unit for applying and checking security permissions. When you build an assembly, you can specify a set of permissions that the assembly needs to run. To ensure then that your code can personally handle all potential security exceptions, you can insert a permission request for all the permissions that your code must have. In this way, you can properly react if the permissions are not granted. The security policy for the assembly is defined at load time based on existing administrative settings and the assembly's evidence and provenience. That an assembly requires a certain set of permissions to run, of course does not guarantee that those permissions will be granted. However, by handling the requests, an assembly can provide specific information to the users.
Another aspect of an assembly's security is code signing. As we saw previously, giving a strong name automatically encrypts the code with a private key. While helpful to ensure name uniqueness and better identify code and vendors, strong name signing does not provide any level of trust.
You can associate a certificate with the assembly by using another Framework tool—signcode.exe. Code signing requires that the software vendor prove identity using a certificate issued by a third-party authority. The certificate is embedded in the assembly and enriches the set of evidences that an administrator has available to decide whether to trust the code.
Strong names and code signing are not mutually exclusive. You can use both to better qualify your code. However, if you plan to use both, apply strong names first.
So you know how to write, secure, sign, and deploy your assemblies. Now we'll look at another aspect of assemblies that I hinted at earlier. Since an assembly is the atomic unit for development, security, and deployment, it must also be the smallest piece of executable code in the .NET platform. Let's see how managed code actually runs.
Aside from Windows XP, which has a modified loader that checks for the CLR header, all other Win32 operating system s (with the exception of Windows 95) load and run managed code by resorting to an old trick. Any Win32 executable can incorporate a stub program that the loader invokes whenever the version of the required operating system does not match the version of the current OS. The required OS version is information hard-coded in any Windows executable. The stub program is normally an x86 executable.
This trick was commonly used by Win16 applications to automatically load into Windows 3.x even if launched from an MS-DOS console. In that case, the stub was given by a simple line that spawned the win.com program followed by the name of the Win16 program.
A similar contrivance let managed applications run under Win32 operating systems. In this case, though, the stub calls into a system function (called _CorExeMain defined in mscoree.dll) that creates and loads an instance of the CLR component. All .NET executables link mscoree.dll, which is a plain Win32 DLL. The stub program that .NET compilers embeds in .NET executable is ignored today under Windows XP and so it will also with future systems thanks to a modified version of the OS loader component. The new loader, in fact, checks for the CLR header in the source executable and automatically starts the CLR whenever needed.
Once the application has become a CLR host, it creates the default application domain (AppDomain for short) for the process and pumps the managed code in it for actual execution. Managed code is inherently safe. Before executing any instruction of managed code, the CLR verifies it. Once again, the verification takes place at the assembly level. The process ensures that no memory is read before having been written, every method is called with the correct number of arguments, every function always returns a value, and more. The code that passed the verification step is said to be type-safe and can be run.
Managed code needs an application domain to execute. An AppDomain provides code isolation as well as security boundaries and unloading capabilities. The AppDomain is the run-time environment in charge of executing the assembly's code. Just as with threads, any process must have at least one primary domain, but additional ones can be programmatically created. There is no particular relationship, though, between threads and domains. They are orthogonal in the sense that an AppDomain can run multiple threads while a given thread can operate on different domains.
An interesting aspect of application domains is that they represent the atomic unit of process isolation. In Win32, security and execution boundaries are placed on the process. And the process, in fact, takes total control of the CPU and the memory space, thus becoming the smallest unit of executable code that can run safely without affecting other applications. Just the verification step discussed a moment ago, though, ensures that in .NET the AppDomain (a subset of the process) can be considered an isolated compartment of running code. In a certain way, the CLR implements a sort of multi-tasking between domains rather than between processes. Anything that happens within an application domain does not affect other domains. Communication between domains is prohibited unless you resort to a dedicated API—the .NET Remoting, which takes care of serializing and deserializing objects across the boundaries.
Speaking of assemblies, it is hard not to mention one of the coolest features of the.NET platform: the ability to mix together components written with different languages. To illustrate, suppose that you have a class written in C# and compile it into an assembly.
Now you can derive a new class from it, irrespective of the language. For example:
Interoperability between compiled components is nothing new. With COM, in fact, you could have clients and servers cooperating irrespective of their native languages. The big difference between the language neutrality in COM and .NET is that .NET provides full support for inheritance, which was completely missing in COM. Cross-language inheritance is only possible if the base class is compiled into an assembly. The reason is that the assembly is made of IL code and this fact straightens out any difference between the languages, applying a sort of source code normalization.
Managed code is compiled to an intermediate language that is not machine code. As such, IL is a language that, at least in theory, can be decompiled. This point poses a serious problem as for the intellectual property of the code you write. Decompiling managed code is certainly possible and some companies are already pushing their own decompiler tools. Other companies, though, are selling tools that obfuscate the .NET binary code so that decompilers can't work.
Decompilers work by recognizing common patterns of code that they "know" to be the typical output of high-level instructions like if or while. Obfuscators scramble instructions, paying attention to produce equivalent code with different binary patterns. Obfuscators are potentially harmful tools because they modify the .NET code. Of course, this does not mean that obfuscators break existing code, but it is theoretically possible that scrambling instructions results in a worse (or improved......) performance.
The truth is that there is no safe and totally reliable way to protect code from decompilation. On the other hand, not all applications really need to invest money to protect code.
Protecting code is a fundamental point only for a small subset of applications and for a portion of their code. For example, the authors of applications that implement critical optimization algorithms may have strong interest to avoid and prevent decompilation. For these same applications, though, obfuscators can be harmful as well: what if the obfuscator scrambling instructions hits the performance? As a final note, consider that obfuscators mainly work by moving instructions around. In doing so, they simply generate new patterns of code. So for ill-intentioned people, decompiling even obfuscated code is not really impossible.
An alternate approach to protect really critical code is implementing it using unmanaged code (that is, plain old C++ code) or isolating it onto a separate machine accessed through a Web service. In this latter case, though, you must be able to provide 24x7 access to the machine. In addition, the latency introduced by Web services must not be a new problem to face for the application. As stated previously, using ngen.exe does not work since you still have to deploy the original (and decompilable) IL code.
For a large number of developers, the assembly has been a concept difficult to grasp at first. Used to thinking in terms of files on disk, for us an assembly defined as a logical group of compiled modules and resources sounds a bit confusing. The confusion can only grow if you then add the notion that assemblies can span multiple physical files. How is it possible that a file is implemented using "more" files?
Aside from the difficulty raised by its official definition, the assembly is a central element in the .NET puzzle. It represents the atomic unit of executable code and the smallest piece of code to which security permissions can be applied. An assembly can be deployed privately or globally and can be made of clear or encrypted code. Assemblies also contribute to solve an old problem that affected Windows for a long time: DLL Hell. By writing applications that use only their own private assemblies you significantly reduce the possibility of conflicting shared files. When you just need to make use of shared assemblies, strong names and the smart internal organization of the global assembly cache prevent DLL conflicts, thus making side-by-side execution a reality.
Dino Esposito is Wintellect's ADO .NET expert and a trainer and consultant based in Rome, Italy. A frequent speaker at popular industry events such as Microsoft TechEd*, DevConnections*, and WinSummit*, Dino writes the monthly "Cutting Edge" column for MSDN Magazine and the "Diving into Data Access" column for MSDN Voices. He also regularly contributes to Developer Network Journal*, MSDN News*, and asp.netPro Magazine*. Dino is the cofounder of http://www.dotnet2themax.com/*
