==================================================================
AN INTRODUCTION TO R:BASE 4.0 & MEMORY MANAGEMENT
==================================================================
PRODUCT : R:BASE VERSION : 4.0
CATEGORY: PERFORMANCE SUBCATEGORY: SPEED
==================================================================
From Dennis Comfort, Microrim Vice President of Development
WHAT IS R:BASE 4.0
==================
R:BASE 4.0 is the latest release of R:BASE from Microrim. It is part
of the Microrim Upgrade Express program and subscribers will receive
it as part of their annual subscription agreement with Microrim.
R:BASE 4.0 is packed full of new features and enhanced capabilities,
but perhaps more important than the features, R:BASE 4.0 sports major
speed improvements throughout the product. We could devote an entire
article just to the new features, and in fact we do later in this issue.
This article will focus primarily on the memory management technologies
which Microrim employed to marshall the speed improvements in this
exciting new release.
R:BASE 4.0 is a 32 bit, protected mode database management system.
As such, it exploits the power and performance of Intel 386 and 486
processors by: a) using the 32-bit flat memory model available on
those processors to ignore previous memory restrictions like 640K
DOS boundaries and segmented memory schemes, and b) using the full
processor speeds of the 386 and 486 machines by executing full 32-bit
instruction sets. The result is a new R:BASE product that runs much
faster than ever before!
When we say 32 bit, protected mode, we are saying a lot. Let's examine
this phrase in pieces to understand what is really different about
R:BASE 4.0 versus previous versions of R:BASE as well as other products
on the market.
16 Bits versus 32 Bits - What's this all about?
===============================================
R:BASE 3.1C and all of it's predecessors and companion products, and
most of the DBMS products that are currently shipping for the PC
platform are all based upon the Intel 16 bit architecture. They are
all designed to run, regardless of how fast, on all Intel machines.
Some 16 bit applications still run on the 8088, others with the 80286,
and progressing through the 80386 and 80486. They use 16 bit data and
memory pathing in the instruc-tions as part of the program. There's
nothing wrong with using 16 bits, and in fact, in all processors below
the 80386, that's the only choice available.
On the 80386 and 80486 processors, a program has the option, given
some significant program changes, of running either in the processor's
real mode (1MB addressable memory), virtual 86 mode (multiple real mode
programs running in multiple virtual V86 machines in the same 1MB
address space), or protected mode (up to 4GB of addressable memory).
In addition, if the program is running in real mode or V86 mode, it is,
by definition, a 16 bit application, meaning that it is restricted in
how much addressable memory can be made available. It will only utilize
the 16 bit instruction set of the processor. In short, you end up
wasting a lot of system facilities (including speed) by only using 16 bit
addressing on a 32 bit processor. Furthermore, if your application
executes in real mode, you restrict how much memory you can use at one
time. This results, for larger programs, in having to constantly swap
code in and out of the real (conventional) memory, causing some perfor-
mance degradations. Of course, you can mitigate how much swapping
occurs and from where by carefully controlling the amount of available
real memory (by using an add-on memory manager), and by controlling
where the code swapping takes place (from disk or from memory,
depending upon what memory manager you use). So, in a real mode
environment, you can get respectable performance, but you are still
running your 80386 or 80486 as a real fast 8088/8086 or 80286. You
aren't utilizing your machine to its full potential.
The chart below shows how R:BASE products use the different processing
modes of the 80386 and 80486 computers. For comparison purposes, we
have also included Paradox 3.5 and the yet-to-be-shipped Paradox 4.0
in this chart.
To use the 32 bit capability of these machines, it is necessary to
modify the product to utilize 32 bit addressing, and then rebuild
(compile and link) these products with 32 bit tools (compilers,
linkers, debuggers, etc). Until recently, there haven't been too many
tools available to do this. Even with tools available, we have been
lacking any operating system which is capable of running these appli-
cations. DOS is a real mode, 16 bit operating system which does not
take advantage of technologies such as 32 bit addressing, protected
modes of the chips, or flat memory models. On the other hand, OS/2 1.X
is a protected mode operating system, but it still runs in 16 bit mode.
OS/2 2.0, recently shipping, is a 32 bit operating system which is just
now gaining some momentum. Windows 3.0 is an operating system of sorts
(which lives on DOS), but also is a 16 bit environment. Windows/NT,
scheduled to ship someday, is a full 32 bit operating system that may
prove fruitful. But today, most of us live exclusively in DOS.
OK, You've convinced me that I'm not getting all I can get from my
machine. How do I get to use my 32 bit protected mode capabilities?
===================================================================
Simple. Use R:BASE 4.0. More good news is that you don't have to do
anything special to achieve this. R:BASE 4.0 automatically does all
of the work for you to use the full potential of your computer. Let's
look a little further into what's behind R:BASE to understand how this
is done.
To build R:BASE 4.0, the development group at Microrim switched from
our traditional use of the Microsoft C compiler family (16 bit) to the
WATCOM C386 32 bit compiler family. This required significant changes
to the R:BASE source code to make use of the 32 bit architecture. The
result is an executable which consists of full 32 bit instructions so
that the speed and architecture of the 32 bit chips (80386 and 80486)
can be exploited. But how do we exploit these chips without a 32 bit
operating system?
Before we can answer this, we need to review a few more concepts. Most
of us using PCs use the DOS operating system. DOS, under normal
conditions, can address a maximum of 640K of (conventional) memory.
This means that our applications must live within the 640K limit.
Without going into detail on why, suffice it to say that two of the
reasons that there is a 640K boundary are: 1) most applications only
use 16 bits to represent addresses, thus restricting ourselves to a
640K boundary, and 2) since DOS is running the processor in real mode,
our application cannot address anything in memory above the one megabyte
threshold.
As you might expect, there are exceptions to the rules. By using add-on
memory managers and DOS 5, we can cheat a little bit and actually
squeeze a few more bytes out here and there, and move device drivers
(such as network and mouse drivers) to some other memory locations
beyond the 640K area. However, for all practical purposes, we, along
with our applications, live within the confines of the 640K memory
barrier. Large applications, like database managers (R:BASE 3.1X is
an example), are far larger than any 640K limit. Therefore, these appli-
cations must use sophisticated memory swapping and overlay techniques
to constantly move program code around within the 640K bounds while the
user goes about his business. The result is performance of products
which could be improved if we stop the swapping.
How do you stop code swapping?
==============================
One alternative is to switch to an operating system which provides a
flat memory model and much larger address spaces (including no 640K
limits) and use them to run our ap- plications. Doing this presumes
that the applications have been altered to run in these environments.
An example of an operating system which provides such a memory model is
OS/2. OS/2 1.X is a 16 bit protected mode operating system. Microrim
ships an OS/2 version of its R:BASE product which therefore is not bound
by the 640K limit. However, it is pretty clear now that the industry is
not going to drop DOS in favor of OS/2 1.X. IBM has recently started
shipping OS/2 2.0, a 32 bit protected mode operating system that also
provides a flat memory model. The acceptance of OS/2 2.0 in the market
is growing, but does not as of now dominate the PC market. In the mean
time, what do DOS application vendors and users do today to get around
these limitations and problems brought on by using 16 bits and real mode
DOS?
Simple. Extend DOS to avoid these limitations.
==============================================
From the user perspective, this really is simple because the user
doesn't have to do anything. All the work must be done by the vendor.
To extend DOS, a vendor (such as Microrim) uses a supplemental product
called a DOS extender which 'extends' the operating system so that appli-
cations can address memory beyond the 640K boundary without having to
deal with the mechanisms or penalties of swapping. There are several
available on the market today. Fortunately, the existence and use of DOS
extenders can be left as an implementation detail for the vendors, and
virtually invisible from a user standpoint. Customers don't have to buy,
load, run, or use DOS extenders. Even better, the customer doesn't have
to load any supplemental memory managers at all (like HIMEM.SYS, EMM386,
QEMM, or 386MAX, to name a few). It is the job of the vendor to modify
their applications to use a DOS extender, and then ship the extender
embedded within their own products. We'll examine how this works in a
moment.
What is a DOS extender and What does it do?
==================================
A DOS extender is consists of a set of program libraries, some of which
reside within the extended application (like R:BASE 4.0), while other
pieces run external to R:BASE as real mode programs. Vendors of DOS
extenders did not want to write an entire new operating system to
replace DOS, but they did have to implement many of the services that
an operating system typically provides so that those services could be
available to pro-tected mode applications.
A typical DOS extender provides the following services to the extended
application:
1) Initialization of operating system and processor registers in
preparation to switch the processor from real mode to protected mode.
2) Service the interrupts from the extended application by providing the
interrupt services or passing the requests on to DOS.
3) The ability to load the protected mode application from disk into
extended memory.
4) Memory allocation/de-allocation functions to create and destroy
memory (buffers) for the application. These functions must be
provided for not only extended memory, but in some cases, also
conventional memory.
5) Core extender code to allow for communication between the extended
application in protected mode and the DOS extender in real mode.
6) Transparent access to DOS and BIOS services via the extender to
obtain basic services such as file input/output functions.
7) Clean up code to terminate the extended application and switch the
processor back into the real mode when the program has terminated.
These functions are provided to extended applications in the form of
both 16-bit and 32-bit DOS extenders. Applications built with a 16-bit
DOS extender have a maximum addressable memory range of 16MB. Applications
which use a 32-bit DOS extender have a range up to 32GB. Applications that
use 32-bit extenders must also be built with 32-bit compilers. R:BASE 4.0
is built using the WATCOM C386 32-bit compiler and the WATCOM version of
the Rational Systems' 32-bit DOS extender, DOS4GW.
This completes the basic tutorial on 16-bit usage versus 32-bit usage
as well as an introduction to extending DOS. Let's proceed with some
specifics on how R:BASE 4.0 operates in this environment.
Loading R:BASE 4.0 - Why does it take so long?
==============================================
When you type RBASE at the DOS prompt, it will take approximately 30
seconds for R:BASE to load and be ready to use on your computer. This
is considerably longer than it has taken in previous versions of R:BASE.
Is something wrong? Why does it do this?
To answer the first question, nothing is wrong. There is a lot going on
behind the scenes after you type RBASE. Before looking at what happens,
we should first understand the typical memory layout of a 80386 or 80486
computer. Refer to figure 1 below.
-->+--------------------+..........
| | | 4Mb
| | |
Extended Memory | |
| | |
| | |
| | |
-->+--------------------+..........
Upper Memory Area -->| | 1Mb
Video Buffers, etc. | |
-->+--------------------+..........
| | | 640k
Conventional Memory | |
| +--------------------+
| | DOS |
-->+--------------------+..........
0k
Figure 1.
This figure shows the basic memory layout of the computer. It assumes
that the only software loaded is DOS, with no special considerations
for loading any drivers or tools high in memory. In this diagram, you
can see that the conventional memory address space ranges from 0K to
640K. It is within this space that DOS typically resides, along with
standard 16 bit applications. (Much of DOS 5 can be loaded higher into
memory, but it is not germain to this discussion and will not be
considered in this article). Above the 640K boundary is a 384K area
called the Upper Memory Area. This space has reserved memory locations
for the video buffers required for the various video cards (for example,
there are reserved locations for monochrome displays, EGA displays, and
VGA displays). However, there are spots within this 384K that can be
made available to DOS or applications. Both conventional memory and
upper memory reside below the 1MB threshold, and therefore is classified
as real memory, because it is the only memory that can be directly
addressed by the processor when in the real processing mode.
Above the 1MB boundary is the computer's extended memory. This address
starts at approximately 1MB, and goes up to the limit of the memory
installed on your machine. Therefore, if you have a computer that has
4MB of physical memory installed, then the first 1 MB is directly
addressable by DOS and DOS applications as real memory, and the
remaining 3 MB is not addressable by DOS or DOS applications unless
a) there are special device drivers loaded to use it (like ramdisk
drivers or memory managers), or b) there is a protected mode DOS-
extended application running, capable of addressing that mem-ory. Note
that there are several memory managers which can be loaded to provide
access to the upper memory area and an area just above that called the
high memory area. We will not discuss those issues in this article, as
their existence is not necessary for an understanding of what is
happening with R:BASE 4.0.
Now that we have an understanding of the general types of memory and
their location, let's see what happens when we load R:BASE (see
figure 2). When you type RBASE at the DOS prompt, DOS loads what it
thinks is the RBASE program from the RBASE.EXE file. In actuality, it
is loading an R:BASE stub program from the front of the executable file.
Step 1 shows the stub and DOS loaded into conventional memory. This stub
is quite small and is always resident until the user exits from R:BASE.
The stub performs a number of initialization functions, including
checking to see whether or not certain command line switches were
present on the RBASE command line. For example, if the user supplies a
-R switch, then the stub will not load the signon screen, and therefore
not load the RSIGNON program. If the user omits -R switch from the
command line (and from the RBASE.CFG), then the stub will load the
RSIGNON program into conventional memory, wait for it to complete, and
then the space occupied by RSIGNON program will be released back to DOS
and the conventional memory pool. (See Figure 2 below.)
->+------------++------------++------------++------------+
| | || || || |
| | || || || PROTECTED |
| | || || || MODE |
Extended | || || || R:BASE |
Memory| | || || || |
| | || || || |
| | || || || |
| | || || || |
-->+------------++------------++------------++------------+
Upper -->| || || || |
Memory Area | || || || |
->+------------++------------++------------++------------+
| | || || || |
| | || || || |
| | || |+------------++------------+
Conventional | || || DOS4GW || DOS4GW |
Memory | | |+------------++------------++------------+
| | || RBASE STUB || RBASE STUB || RBASE STUB |
| +------------++------------++------------++------------+
| | DOS || DOS || DOS || DOS |
->+------------++------------++------------++------------+
Step 1 Step 2 Step 3 Step 4
Figure 2.
Once the initialization is complete, the stub program loads the DOS
extender. We are using the DOS4GW DOS Extender, a special version of
the Rational Systems, Inc. DOS Extender distributed by WATCOM, Inc.
The DOS4GW.EXE is a separate file on your distribution disks. Step 2
shows DOS4GW loaded above the R:BASE stub in conventional memory. As
previously described, there are many important functions which must be
performed by the DOS extender. Once the initialization functions are
completed by DOS4GW, it loads the protected mode version of R:BASE (also
located in this same RBASE.EXE file) from disk into extended memory.
This will take several seconds to complete. Once done, there are some
final initialization steps, followed by a command by DOS4GW to switch
the processor from real to protected mode. Step 3 shows all components
loaded in their respective memory areas.
It should be clear at this point why it takes longer to load R:BASE 4.0.
We have executed several other steps which previously were not necessary
in the 2.X and 3.X versions of R:BASE. In addition, we are doing a
complete load of a 2MB executable, which never was required before for
the DOS R:BASE products.
With all software loaded, you are now ready to use R:BASE.