At the beginning of the 90’s, the PC platform was often mocked by its rivals. PCs of that era had become much more powerful than, say, an Amiga 500. But the Amiga offered a flat memory address, while a DOS program could only access memory using cumbersome 64 KiB segments. And to add insult to injury, there was this strange 640 KiB memory limitation. No matter how much physical memory you had in your box, the utter most important Conventional Memory was limited to 640 KiB!

The Legend teaches us that Bill Gates once declared that “640 KB ought to be enough for anybody“, then designed MS-DOS to enforce this limitation.

The truth is of course a little more complicated than that.

 

An IBM PC legacy

Segmented memory

The first “IBM PC”, numbered 5150, was designed around the Intel 8088, a castrated version of the 8086. As a 16 bit CPU, it can only manipulate addresses up to 64 KiB in its registers. Thus, in order to access more memory, a bigger address space is split into multiple segments.

The way the 8086 generates addresses from segments is really peculiar. Instead of using a register to identify a segment and another one to indicate an address inside it, a new segment begins every 16 bytes. As segments are 64 KiB long and there is no memory protection, it means that segments overlap. Any physical address can be accessed by up to 4096 segment+offset combinations! This also caps the maximum addressable memory to around 1 MiB, corresponding to the 20 bit address bus of the 8086/8088.

x86 Real Mode Segmentation

x86 segmented memory model is quite peculiar: segments can overlap!

The 640K ceiling

With its 8088, the IBM PC could address up to 1 MiB of memory. A huge amount in 1981.

This address space had to be shared between the RAM, the video memory, and the various peripherals. IBM decided that the lower 640 KiB would be reserved for the (user accessible) RAM, while the upper 384 KiB would be used to address the video memory, the BIOS and the peripherals. These two zones should be to be big enough to fulfill their purpose…

And Microsoft? It never imposed any limitation!
Take for instance the almost forgotten Apricot PC, which was another MS-DOS machine of the time, albeit incompatible with the IBM PC. It puts the frontier elsewhere and the user accessible RAM could be sized up to 768 KiB.

So the infamous 640K memory ceiling was due to IBM! And also a bit to Intel…

 

The 286, a “brain dead chip”

In 1984, IBM commercialized the IBM PC/AT (“Advanced Technology”!) based on an Intel 80286. The 80286 is of course fully compatible with the 8086. It is still a 16 bit CPU relying on segmented memory, but it can access to up to 16 MiB of memory. And it provides some form of memory protection.

But…

In order to retain compatibility with the programs and OSes written for the 8086, all these shiny features are only accessible when running in the newly introduced Protected Mode. When operating in Real Mode (a.k.a “Legacy Mode”), the 286 manages memory just like a (faster) 8086. No memory protection and no access to more than 1 MiB of memory!

The Real Mode is the default operating mode of the 286 when starting up. Once loaded, the OS can switch to Protected Mode and let the user reap the benefits. Alas MS-DOS, along with its drivers and its existing applications, had been written when no PC came with anything else than a 8086 or a 8088. Unfortunately, their developers made some bold assumptions on that matter. Modifying MS-DOS to operate fully in Protected Mode would have been an extensive work and would have broken the compatibility with most applications and drivers!

A solution might have been to switch back and forth from Protected Mode. But entering Protected Mode on the 286 is a one way ticket: if you want to switch back to Real Mode you have to reset the CPU! Bill Gates, who understood very early all the implications, is said to have called the 286 a “brain dead chip” for that reason.

Hacking the way to more memory

Fortunately, some (rather hacky) schemes were devised to allow DOS programs accessing more memory. Otherwise, the PC as a computing platform would have failed…

Understanding these schemes requires some naming conventions.

  • Conventional Memory is the precious first 640 KiB of addressable memory. In MS-DOS the instructions of a program shall reside in it. That partially explains why some latter programs / games required most of conventional memory to be at their disposal. And that is why freeing a mere KiB of it was a worthy battle.
  • Upper Memory Area refers to the area that was reserved by IBM for the video adapter and the various peripherals. Note that this area is only reserved and is not always entirely consumed!
  • Extended Memory is the memory out of reach of for a 8086/8088 (past the first MiB). It is only naturally accessible on latter Intel CPUs. This is the memory pool which access is required for more advanced computing purposes.
IBM PC Memory map

IBM PC Memory map (by Wtshymanski, CC BY-SA 3.0)

HMA (High Memory Area)

In Real Mode, memory segments start every 16 bytes and are 64KiB long. So the last segment exceeds the 1 MiB memory limit by almost 64 KiB!

On a 8086 this is a non-problem. Indeed, its 20 pin address bus makes impossible to access more than 1 MiB of memory. All generated addresses that exceed 1 MiB are aliased to the very first addresses.
But the 286 is equipped wit a 24 pin address bus. So it can access this memory block, even in Real Mode!

This very last segment is called HMA (High Memory Area). This caused compatibility issues with some 8086 programs that used the last segment, expecting it to loop back to the first addresses. But it also allowed HIMEM.SYS to load some parts of MS-DOS 5 to HMA on a 286/386/486, freeing up to 46 KiB of precious conventional memory.

EMS (Expanded Memory Specification)

In 1985 Lotus, Intel and Microsoft announced the EMS standard, allowing a standardized access to extended memory.

A 64 KiB unused and contiguous portion of Upper Memory is identified and split into four 16 KiB segments. A hardware assisted bank switching mechanism and an appropriate DOS driver are then used to access any location in extended memory when requiring access to those four segments.

It potentially gives the applications access to the whole 16 MiB address space, even in real mode.  Although, programmers need to manually configure the four 16 KiB “views” via the driver.

EMS Illustrated

EMS Illustrated: a free 64K block from Upper Memory gives a “view” to four 16K pages from Extended Memory.

Great! Is there any drawback? It requires dedicated hardware support on the motherboard and the use of a driver consuming a few KiB of previous conventional memory. And programmers cannot easily manipulate data structures bigger than 16 KiB.

Quoting Bill Gates himself:
“It’s garbage! It’s a kludge! … But we’re going to do it!”

The dedicated hardware often takes the form of an expansion board that can also by used by older 8086/8088 machines.

XMS (Extended Memory Specification)

Another scheme exists to access extended memory: XMS. The idea: switch the CPU into Protected Mode and copy segments data from/to extended memory to/from unused segments of upper memory, then to switch back to Real Mode to allow a DOS program to access it.

The advantages? It does not require any dedicated hardware. It also frees the programmer from being caped to 16 KiB of continuous memory.
The drawback? The 286 requires a reset to switch back to Real Mode, making the use of XMS on a 286 extremely slow! Thus it was rarely used, until later…

EMS and XMS both require free blocks in upper memory. If there are not enough free blocks available for both to be available, the user has to choose which standard he will enable in the boot configuration files. His judgment will be based on the programs he intends to launch. Configuring MS-DOS on a 286 was not for the faint hearted!

HIMEM.SYS, which manages the access to HMA, is also in charge to provide an access to extended memory.

To be fair with the 286, a clever method was devised in latter revisions of HIMEM. It allows accessing XMS without requiring a switch to Protected Mode. This method is built around LOADALL, an undocumented 286 instruction. This instruction manipulates the entire processor state, including “hidden” registers that are out of reach of normal instructions. A hidden segment register exists among them, which can be modified to access memory addresses beyond 1 MB, even in real mode.

This trick is only used on the 286 as latter CPUs do not require it.

Enter the 386

When it designed the 80386, Intel now understood well that 8086 and DOS compatibility would be its main asset. Neat features were also added.

  • A MMU (Memory Management Unit) which performs “Virtual Memory” management, mapping virtual addresses into physical space. This opens the way for modern OSes such as Linux.
  • The Virtual 8086 mode, a sub-mode of Protected Mode, in which Real Mode program runs in a kind of virtual machine. The monitor itself operates in protected mode so advanced memory usage such as paging and isolation are possible.
  • An instruction to quickly switch between Protected Mode and Real Mode on the fly.

XMS is now easily usable on a 386!

The 80386 still relies on a segmented memory model. But being a 32 bit CPU, the segment can be 4 GiB long, giving the same facilities as a flat memory model.

EMM386.EXE, the memory manager introduced with MS-DOS 5 in 1991, locks DOS itself into a Virtual 8086 task. It also provides DOS with a driver emulating EMS, using extended memory from Protected Mode. This driver makes use of the 386’s MMU to map blocks into the “EMS Window”: no memory copy occurs. Thus, EMS memory is faster than XMS. But it is still awkward to use.
EMM368 is also able to load device drivers and TSRs into Upper Memory Area if enough free segments are available. By using LOADHIGH and DEVICEHIGH directives in autoexec.bat and config.sys, instead of their older counterparts LOAD and DEVICE, it is possible to free dozen KiBs from the conventional memory!

To be fair, those techniques were first introduced by Quaterdeck in its QEMM memory manager, then by Digital Research in its DR-DOS 5, before being mimicked by Microsoft in MS-DOS 5.

DOS, extended!

With EMS, XMS and ways to free most of conventional memory, PCs with MS-DOS became serious contenders in the video gaming platforms arena. However, accessing EMS and XMS is a slow process. And, even on a 386, they don’t allow to get rid of the 16 bit memory segmentation schemes.
Amiga users still had many arguments to mock PC users… until the generalization of DOS Extenders, which finally allowed the PCs to fly. One of the first games making use of a DOS Extender was no-less than DOOM!

DOS4/GW when launching Doom

When you saw DOS4/GW, you knew you were in for some serious gaming. If you have ever wondered what that DOS4/GW is, it’s a DOS Extender!

DPMI (DOS Protected Mode Interface)

One major software editor impeded by the 640K memory limit was Microsoft itself! Windows 3.0 was an ambitious project, but it required more memory and some facilities that Real Mode was unable to provide. Thus Microsoft introduced DPMI in 1989, before giving its control to an industry committee.

DPMI describes a collection of services to be offered by a DPMI host, which allows client applications, called DMPI clients, to run in protected mode while still being able to make use of DOS services in a well-behaved and hardware-independent way.
In order to fulfill its role, a DPMI host has to perform mode switching, allocate extended memory, allocate conventional memory, control the interrupt subsystem, communicate with real mode programs, and read or write certain CPU control registers.
Running in Protected Mode, some form of multitasking between DPMI clients is possible while, for compatibility sake, Real Mode TSRs are still present alongside!

Although it is quite possible to code a DPMI host on a 286, most of them require a 386. Thanks again to its fast mode switching and its Virtual 8086 mode!

DOS Extenders

The DPMI specifications allowed the apparition of many DOS Extenders, which are DPMI clients that load and take control of an application. The purpose is to service them with an extended version of the APIs that DOS and the BIOS provide via software interrupts (respectively 0x10 and 0x21). In order to avoid switching back to Real Mode too often and save performance, DOS Extenders also handle frequent hardware interrupts, such as those generated by a mouse.

With a 32 bit DOS Extender, a 386 is able to run true 32 bit programs and to allocate huge blocks of memory. At last!
This also ends the time when the conventional memory pool was the fastest available. Applications built on DOS Extender thus require far less free Conventional Memory, easing the painful process of crafting custom autoexec.bat and config.sys in order to run ones favorite games.

Games running in Protected Mode

DOS Extenders finally allowed games to run in 32 bit Protected Mode, making at last good use of the powerful Intel CPUs.

While still a little bit hacky, DOS extenders finally allowed PCs to make good use of their memory and powerful CPUs. A few years later, Windows 95 made those memory headaches a thing of the past as MS-DOS became progressively irrelevant.


11 Comments

Fabien Sanglard · June 10, 2018 at 05:27

I wonder where you found the inspiration for these quotes….

    XtoF · June 10, 2018 at 10:14

    * “640 KB ought to be enough for anybody” is of course not from Bill Gates. But it was attributed to him for decades and one of the point of this post is to debunk it.
    * “It’s garbage! It’s a kludge! … But we’re going to do it!” according to Wikipedia (sourcing a PC Magazine from 1985) he told that during a COMDEX. I admit this one is dubious. I will make a modification to appear less authoritative.
    * “brain dead chip” is drawn from the book “Microprocessors: A Programmer’s View” (McGraw-Hill). This one is also dubious, that’s why I use the conditional: Bill Gates, […], is said to have called the 286 a “brain dead chip”

    The purpose of these quotes is to ease the reading for a person who is not already well versed into this subject. Of course, I was not standing along Bill Gates during the 80s, so I cannot testify their veracity first hand…

VK2FVAX · June 11, 2018 at 02:59

Going through the whole era and doing far too much asm to ensure virginity, I was most amused by your “more powerful chip” the 8086 vs Amiga’s 68000. Granted they were a few years apart, but the original 68000 ran at a much faster cycle time, yes it had a pseudo flat memory model (I’m being intentionally pedantic) ..it had 8* Data Registers and 8* Address registers. Lets compare this to the appallingly register starved 8086/8088 which has 4x GPR’s, then the segment and pointer regs SS, ES, DS, BP, DI, CS, BP, and SP. Given the 68000 registers were all 32bit internally on a 16bit databus compared to the 8086 which was 16bit internal, I find your statement of “more powerful than an Amiga 500” to be in error. Perhaps you’ve not experimented seriously with the A1000/A500 and compared them to PC’s of the era. That’s my 5c worth.

    XtoF · June 12, 2018 at 00:59

    You’re not the first one to mention this sentence, and you may have misunderstood my intent. Of course I am not comparing a 68000 @ 8MHz against a 8088 @ 4.77Mhz. The x86 chip would be no match against the 68K. I am referring to the early 90s, so I though “386 & 486”. As I said in another reply, I concede that the 68000 ISA is far superior to the 8086 ISA. BUT, a 386 is in another league. Yes the 68000 has internal 32 bit registers, but is build around a 16bit ALU and a 16bit BUS. So I’m quite sure you avoided as much 32 bit operations as possible when you coded ASM 😉
    But I will modify that sentence. I don’t want to give the impression that I denigrate the Amiga. Because I love this machine!

Osmodious · June 11, 2018 at 16:32

Wait…”Of course, PCs were much more powerful than, say, an Amiga 500.” What? An Amiga with a 68000 (7.14mhz, 16bit) and custom chips for video, sound and animation versus a PC with an 8088 (4.77mhz, 16 bit w/8bit bus) and no built-in anything but available sound cards (after 1985) and video (CGA, which was far inferior, for reasonable money and EGA starting in 1984 which was still inferior and cost as much as the PC itself)….how is that more powerful? PC ATs, also out around 1984, with 80286s were fully 16bit but only had 6mhz, then upgraded to 8mhz later (and still had no other built-in handling of sound or video). Heck, the memory handling alone made the Amiga a more powerful machine. That it was able to be targeted at both the consumer market (A1000, A500) AND business market (A2000) with the same capabilities (expandable with the A2000) was remarkable.

    XtoF · June 12, 2018 at 00:52

    In that sentence you quote, I speak about PCs from the early 90s. So you can discard the 8088…
    I currently own an Amiga 500 and I know this machine very well. Any 386 (they start at 16MHz) is way more powerful than a 68000 at 8MHz. The ISA of the 68000 is quite neat and far superior to the x86’s (which is also older). But don’t be mistaken by the 32bit registers, it’s a 16bit CPU at its core. So 32 bit operations are very slow. And it’s also slow accessing memory.
    But, what I point in this article, is that despite its apparent superiority, even a powerful 486 is crippled by all the legacy dating back to the very first PC.
    Concerning the custom chips, I agree with you that they help the Amiga a lot. For 2D scrollers at least. Because they aslo were the bane of the machine when Doom-like became popular. Thanks to the bitplanes a stock Amiga 1200 was unable to run any decent clone of Doom properly.

    I always find post like yours hilarious. You evidently had an Amiga in your youth and loved it. That’s fine and respectable. I also love this machine. But 20 years after the demise of Commodore, please don’t rehash me the very same baseless arguments that I could read in the biased magazines I bought when I was 12. The machine was an engineering marvel, but proved to be a dead end.

      Matthew Smith · June 12, 2018 at 14:58

      My family had an Amstrad PC1640 which we bought in 1989 or so, and although it was introduced in 1986 there were plenty of them still being sold in the early 90s. (Not sure when they were axed; Amstrad brought out the PC3086 in 1990 which still used the 8086.) Most PCs being sold in the very early 90s were still 8086-based and much less powerful than the Amiga although they had more expansion potential than an Amiga 500 or 600. By about 1993 or 94, yes, those machines were over the hill and our next computer was a 486 which we bought in 1993 or so.

Andrew · June 12, 2018 at 19:41

Nice article, well written! You have a factual error in your description of EMS. It did not access extended memory at all, but rather it gave access to memory on an add-in card. This type of memory was available on 8086-based PCs as well, and was quite rare. The only people I knew that used it were using Lotus 123 as a small database and needed several megabytes memory to hold their enormous tables!

    XtoF · June 12, 2018 at 20:24

    Thanks!
    Concerning EMS, you’re right. That’s why I said that it required dedicated hardware. I intended an expansion board, but if it is not clear I may have to rephrase this paragraph 🙂 You also have a point mentioning that it was also available to 8086, I think I should mention it. In later machines EMS was emulated. Thus it became a scheme to access “normal” RAM.
    The difficulty for me is that I want to explain why memory management was cumbersome in the early 90s. It’s hard to be exhaustive while not loosing the reader 😉

Olivier · July 31, 2018 at 12:49

Gold.
Pure gold.
Thank you very much for this excellent post.

Sobre 640K (de memória) | Retrocomputaria · July 20, 2018 at 12:01

[…] Ah, os 640K de memória. Um dos símbolos da era do MS-DOS. Deveria ser o suficiente para todo mundo, mas não era, especialmente com planilhas imensas e mundos de Doom. E por isso a história é um pouco mais complicada do que parece. […]

Leave a Reply

Your email address will not be published. Required fields are marked *