Microsoft's Windows NT operating system, like most other operating systems, provides a set of services to processes running under it. Windows NT arguably comprises an operating system similar 6 to the Unix family of operating systems. For purposes of comparison, the designers of Unix and other Unix-like operating systems (e.g Linux and NetBSD ) defined these services in terms of C programming language functions 1 One refers to the set of these C language functions as "system calls". Programmers working with Unix and Unix-like operating systems often make a distinction between a "library call" and a "system call" A system call necessitates that the operating system kernel do some work. A library call doesn't necessarily cause the operating system to do something.
Users of mainframe operating systems like IBM's MVS or HP's VMS refer to the moral equivalent of "system calls" as " system services". The designers of mainframe operating systems usually defined the services they provided in terms of (possibly special) assembly language instructions and CPU register contents.
Microsoft calls the services that NT provides an Application
Programming Interface or "API" Microsoft describes most of this API
in terms of C or C++ functions, but they make no distinction between a
straight-through call to an OS function, and a call to a library
function. Microsoft chose to blur the boundary between user program
and operating system. The designers of NT don't describe all of the
services that NT provides as C or C++ functions. For example,
"Structured exception handling" appears in Win32 programs in the
form of reserved compiler keywords that possess very special
semantics. Catching a memory access violation involves the NT
kernel finding which exception handler to call
([2], pp 92-96), so the __catch
compiler keyword becomes part of the NT API.
I list below some web-accessible references to documents that state openly that NT's real API remain undocumented.
I know that I take some semantic liberties in conflating the word "undocumented" with "hidden" I believe that the amount of effort required to reverse-engineer an API that exists only as compiled object code renders "undocumented" and "hidden" equivalent.
Note that I emphasized the admissions in the text below.
One of the most interesting things about the NT API is that it has never been comprehensively documented by Microsoft. This must make NT the worlds only commercially available operating system with an undocumented set of native system services!
[...]
A neat facility available through the native NT API but not through Win32 is the ability to cancel all I/O requests that you have outstanding on a particular file handle.
This API, which is called the Native API, is almost entirely hidden from view, with only a handful of its function documented in generally accessible publications.
The NT system-call interface, called the Native API, is hidden from programmers and largely undocumented. The number of UNIX system calls is roughly equal to the number of Native APIs, around 200 to 300. The API that UNIX applications write to is the UNIX system-call interface, whereas the API that the majority of NT applications write to is the Win32 API, which translates many Win32 APIs to Native APIs.(From page 2 of the article.)
I assume that the "translation" mentioned above does not happen without some extra execution time cost.
The Native API is equivalent to the system call interface on traditional monolithic operating systems such as most UNIXes. On most UNIXes, however, the system call interface is well documented and is generally available for use by standard applications. For example, the read() call for reading data from a file, socket, or input device in most flavors of UNIX is a system call that is handled by code in kernel mode. In Windows NT the Native API, its system call interface, is hidden from programmers behind higher level APIs such as Win32, OS/2, POSIX or DOS/Win16.
Vogels cautioned that NT consists of millions of lines of code and only six pages of documentation, which barely tells you how to get it off the CDs and how to build a kernel from scratch at the top level. You need a 5GB disk (I managed easily to fit it in less than 3GB along with the entire build environment). This amount of code makes it difficult to search for strings;
{findstr /s /I}doesn't really cut it when you're talking millions of lines of code. The MS Index server, although excellent for Web pages, crashes on source code; and when it works, it comes up with 1,000 irrelevant matches.The conclusion was that Vogels's group used source code only as documentation (there is no other documentation for NT), examples, and to understand the behavior of NT. It turned out to be useful for debugging, and it led to the discovery of interesting APIs that are not documented or available in Win32.
As development of NT 4.0 began in 1995, Microsoft invited a Diskeeper. developer to Redmond, Washington, to participate in the design and implementation of NT 4.0's defragmentation support. Basing Diskeeper on NT's built-in support let Executive Software avoid shipping custom versions of NT and meant Microsoft's technical support didn't have to troubleshoot non-standard versions of NT. One month before NT 4.0's public release, Executive Software offered a trial version of Diskeeper 2.0--the version for NT 4.0--on its Web site. (For Jonathan J. Chau's review of Diskeeper 2.0, see Lab Reports, "Diskeeper 2.0," April 1997.) Symantec has since entered the NT disk defragmenting market with its Norton Utilities Speed Disk, which also uses the NT 4.0 defragmentation support.
The defragmentation support introduced in NT 4.0's file systems consists of five commands: GetVolumeBitmap, GetRetrievalPointers, and MoveFile (common to both FAT and NTFS), and GetVolumeData and ReadMFTRecord (specific to only NTFS). Microsoft doesn't document these commands or officially acknowledge that they exist.
This article appears in a slightly different version at the Systems Internals web site.
- Windows NT System Services (or executive system services)
Undocumented functions callable from user mode. For example, NtCreateProcess is the internal system service the Win32 CreateProcess function calls to create a new process.
I believe that all this evidence proves that Microsoft hides APIs for the Windows NT operating system. With that conclusion firmly established, all that remains is to decide what the consequences of this unique situation might be.
Note that in this analysis, I don't attribute anything special to the NT Native API other than a (maybe substantial) speed advantage. I feel that experience with OS kernels that possess a structure similar to NT (Mach, used in both OSF/1 and NeXTStep) allow anyone to say that two extra context switches per system call imposes a distinct time penalty4.
I also only want to examine consequences of NT's secret API. One could possibly impute a motive based on this sort of analysis.
Win32 calls remain the officially encouraged, publicly documented interface to the NT kernel, yet they go through a (possibly time-consuming) extra layer of indirection (the Win32 protected subsystem) to get there. The indirection can cause a substantial execution time difference. Harvard researchers and IBM, have documented this difference. Some functionality can only be gotten at through a native NT system call: cancel an outstanding asynchronous I/O request, create a process with a cloned address space, or get a list of DLLs that various processes have loaded. You can't examine various system properties like clock frequency, memory configuration, number of processors, list of active page files or list of open handles in all processes using Win32 calls. 5
This means that if a third party develops something for NT without the blessing of Microsoft, they must make a difficult decision:
Use only a documented interface (Win32, POSIX, etc) resulting in a slower product.
Make a faster product by using an undocumented interface (the real NT system calls). This means doing a lot of work to reverse-engineer 5 the NT system call interface (for other copies look at alternative copy 1, or alternative copy 2).
If the product becomes popular or makes money, Microsoft can make a faster competing product using the real system calls, or they can change the real NT system calls out from under your product at the next release of NT. In either case, Microsoft can cause their competing product to inherit your market.
Industry analysts acknowledge this:
But Microsoft's support of Mono is simply the same old same old for the software giant, IDC analyst Dan Kusnetzky said. Microsoft has historically achieved market dominance by controlling APIs, and forcing competitors to write software to its APIs, only to turn around and change those same APIs. "Instead of satisfying their own customers' demand, competitors are busy catching up with Microsoft," Kusnetzky said. "It looks like they've gotten someone in the open source community to play the game of following Microsoft around and trying to do what they do."
The example of disk defragmentation API above illustrates another potential consequence of an OS with undocumented system calls. One software vendor in some manner obtains special dispensation: advance knowledge of the secret API. This gives that vendor such an enormous benefit that few others enter that particular market.
In other words, Microsoft can pick and choose which vendors receive a low-cost entry into the market for a particular piece of software, and which vendors must pay a high price (probably in reverse-engineering time) for entry into that same market.
Microsoft could deny all programmers outside of its employ explicit knowledge of an API, or repeatedly change an API with each new version of the NT operating system. In effect, Microsoft could reserve an entire software market for themselves alone.
The general effect remains the same: nonexistant or very limited competition in a market for a particular product. In other markets, this results in lower-quality products at a higher price, or in lack of innovation. I doubt any particular PC software market will act any differently.
Netscape experienced this exact phenomenon. According to an October 8, 1998 New York Times article:
And Microsoft, the people added, did what it has always denied it does -- used access to its technology as a powerful lever in business negotiations, by offering Netscape preferential access to the Windows "application program interfaces," or A.P.I.'s, the links that enable other companies' programs to run smoothly on the Windows operating system. By turning down the deal, Netscape, they say, would not have that preferred access to Microsoft technology -- a threat that Microsoft fiercely denies making.
The New York Times requires you to provide some demographic information when you register for access to the NYT article.
One other effect might come about. Real Gurus might re-emerge.
Before the rise of the mass-market computer, Unix and freeware operating systems, every hardware vendor provided their own operating system. These systems nearly always had weird little undocumented features and commands. People who worked or previosly worked for for a particular vendor often possessed knowledge of the that vendor's operating system and hardware that others did not. They were real Gurus. Currently, one can become a Guru for a freeware OS by merely working hard. Studying and hacking on the source code suffices to become a freeware OS Guru. By nature of their source code distributions, the freeware OSes possess no deliberately hidden knowledge, although some knowledge may seem obscure due to poor or non-existant documentation.
In combination with the "favored vendor" effect, a hidden API could mean that some select few will become Real Gurus, possessors of secret knowledge. Those insiders will end up fulfilling a role similar the Masters of old-time Guilds. They will govern who enters the inner circle, who gets to know and use the secret knowledge. We could end up in a state similar to that of the medieval Guild systems: fractured, incomplete knowledge jealously guarded by a select few causing a general stagnation and lack of innovation.
The Software & Information Industry Association (formerly SPA, the Software Publishers Association) agrees with my conclusions. Pages 12 to 15 of the document Competition in the Network Market: The Microsoft Challenge, dated June 19, 1998, contain a section titled Hidden APIs. Here are two quotes that I find especially illustrative. The first quote illustrates the speed advantage that a programmer can derive from hidden API functions:
For example when Microsoft released Internet Information Server (IIS), it significantly outperformed Netscape Server on the NT Platform. Microsoft insisted that its developers had not had any additional acceess to information than had Netscape developers. Yet after careful review, Netscape developers were able to utilize previously undisclosed information about NT in their own products. Future releases of Netscape Server were competitive with IIS in subsequent testing.If you write programs using a documented API, the programs run slower. The second quote illustrates that Microsoft uses the hidden APIs to make their applications the best in any particular market:
Microsoft can write application code that can run optimally on an operating system, has advance knowledge about future releases, knows which programming method to choose over another, and can tweak the OS code prior to final relase to advantage3 its own applications.If you perform the costly task of reverse-engineering the hidden APIs in order to compete with Microsoft, they change those hidden APIs to favor their products.
The SPA document illustrates precisely the Hard Decision described above.
Need more confirmation? Notorious Windows book publishers IDG put out a slim (335 pages) volume that "documents" a lot of NT internals, including the native API. Conservatively, 6 man-years of effort went into reverse-engineering all this material. This confirms the massive effort required to at least partially document NT's secret API.
Undocumented Windows NT
Prasad Dabak, Milind Borate and Sandeep Phadke
IDG Books, 1999
ISBN 0-7645-4569-8
Although Microsoft Windows NT is one of the most popular operating systems in the corporate world, no book has documented what actually goes on under the hood - until now. Undocumented Windows NT dissects the Win32 interface, deconstructs the underlying APIs, and deciphers the Memory Management architecture to help you understand operations, fix flaws, and enhance performance.
In this groundbreaking guide, three experts share what they've dug up on NT through years of hands-on research and programming experience. The authors' in-depth investigation uncovers both the strengths and the weaknesses - and reveals how you can make any Windows NT system more stable and secure.
I had a look at this book in a local bookstore on January 26, 2000. It possesses a lot of the flaws of a typical "Windows technical book": pitiful indexing, obvious lack of editing, and very little exposition on anything other than the syntax of the programming question at hand. The authors devote an entire appendix to prototyping and describing the NT native API functions' syntax. Material in that appendix apparently didn't make it into the book's index, nor are the function prototypes listed in alphabetical order. If you buy it, you will doubtless encounter a lot of frustration along with the information.
That said, the book contains a truckload of documentation on functionality only hinted at elsewhere.
Another book covering the NT "native" API just came out:
Windows NT/2000 Native API Reference
Gary Nebbett
New Riders Publishing, February 2000
ISBN 1578701996
I had a quick thumb-through of a copy of this book. It looked pretty good, given that it listed the native API functions in alphabetical order, and covered the non-Win32 uses of some of the functions.
[1] Several issues hide in that: whether programmers and designers best describe OS services in terms of C functions (or any function-like interface at all), or some elaborate assembly-language specification (like VMS) that allows almost any programming language to use the OS services.
[2] Inside Windows NT, 2nd Edition, by David A. Solomon, Microsoft Press, 1998, ISBN 1-57231-667-2.
[3] Verbing weirds language. "To advantage" reads poorly.
[4] According to a paper describing Linux on a Mach microkernel,
Early efforts to layer OS personality servers on top of the microkernel have had disappointing performance due to the extra message-based communication between the system components. Often, as much as a 40% performance cost has been reported.
[5] Inside Windows NT System Data by Sven B. Schreiber, Dr Dobb's Journal, Nov 1999, pp 40 - 49, for an account of how Schreiber reverse engineered the NtQuerySystemInformation() native system call.
[6] Analysis
of the Security of Windows NT
by Hans Hedbom, Stefan Lindskog, Stefan Axelsson, and Erland
Jonsson
From Section 8.2, "UNIX versus NT", page 65:
Even though Microsoft Inc. would like to have us believe otherwise, NT does not in fact contain many ideas that UNIX has not either pioneered or picked up during the seventies or early eighties. Both systems have from a research perspective a distinctive seventies feel to them. Especially from a security perspective one is struck by the similarities between the respective systems. There are in fact many more similarities than differences.