S Y M B O L T A B L E M A N A G E M E N T
S Y M B O L T A B L E M A N A G E M E N T
by
Riley Paulk
The Author
of
ClipCASE
Out Of Memory Continue [Y/N]
If you are a Clipper Developer and you have written anything
approaching a medium to large application in S'87 then you have
probably had occasion to see the message above. One is espe-
cially intrigued by Nantucket's recovery option "Continue [Y/N]"
-=- what they should have said is "Continue [Y/N/Well Maybe, But
I don't Think So]". Memory management problems have been a thorn
in the side of Clipper Developers for as long as they have been
writing Clipper Applications. Roger Donay of dClip fame says in
an article he wrote on Clipper Linkers "If you ask the Clipper
programmer about his biggest frustrations, he/she will virtually
always answer the same way - memory usage and development time."
This writer considers memory management problems to be something
like an albatross hanging around his neck. The following infor-
mation is provided in the hope that it will help others through
the process of solving these problems.
This information should be of use to you if:
1. You are still writing applications in Clipper S'87
Code.
2. You are maintaining applications that were written in
Clipper S'87 Code.
3. You have S'87 applications that you are considering
investing the time necessary to port them over to
Clipper 5.0(1).
4. You have applications in which you had to use Nan-
tucket's SWITCH or any number of third-party SWAP rou-
tines for clearing up enough RAM to allow your applica-
tion to run in that little 640K box we all have to live
in.
5. You use compilers such as WarpLink, Blinker, dClip with
a myriad of pre-processors, dynamic overlays, and other
magic aimed at dealing with memory management problems.
6. And, maybe, even if you have already made the quantum
leap to Clipper 5.0(1), are writing exclusively in
5.0(1) and have nothing to do with S'87.
The number one perpetrator of memory management problems in
any Clipper application is the Symbol Table. With well managed
overlays whether static, as with executables produced using
Plink86 Plus, or dynamic overlays produced with dClip, WarpLink,
Blinker, or RTLink 5.0 a tremendous amount of code can success-
fully be incorporated into a Clipper application. The problem is
that no matter how well the overlays are managed the Symbol Table
keeps growing.
A topic so dear to the heart of Clipper programmers as this
has already been addressed extensively by numerous authorities in
the field. The documentation that accompanies Steve Steiner's
excellent memory management tools library called SmartMem and a
text file available on many Clipper oriented BBS's called CLI-
PLINK.ZIP written by Roger Donay of dClip are both especially
informative on the subject of Symbol Tables and the management of
problems related to Symbol Tables. Rather than covering the same
material again in this article, it is recommended that, if you
are not already familiar with Clipper's use of Symbol Tables, you
read those documents or other sources of similar information.
In brief, memory variable references as well as references
to procedures and functions, including those in Clipper itself
and its extend library, are held in the root overlay area [that
portion of the executable that stays in RAM throughout execution]
in an area referred to as the Symbol Table. Essentially each
uniquely named "symbol" in a logical overlay area requires 22
bytes of RAM. Therefore, as your application grows, so does your
Symbol Table. The code for your application can be split into
manageable size parcels either through static overlays or the use
of dynamic overlay techniques but the Symbol Table must remain
intact and must, in most cases, remain in the root area. Unless
corrective action is taken, this Symbol Table will eventually
become the limiting factor on the size -=- read that power -=- of
your applications.
There are many techniques available for coping with the size
of the Symbol Table. The first, and obviously most effective,
technique is don't create the symbol to begin with. Use con-
stants instead of memory variables. A simple example of this
technique is:
t = 10
l = 20
b = 16
r = 59
@ t, l, b, r BOX "ÉÍ»º¼ÍȺ "
Which is a perfectly ridiculous way to accomplish:
@ 10, 20, 16, 59 BOX "ÉÍ»º¼ÍȺ "
If you use the first method to draw a box in each of fifty
code segments compiled into twenty-five object files with two
code segments in each object file you will have wasted 2.2K of
RAM. Remember that it is uniquely named symbols within an object
file that counts. Change the variable names to top, left, bot-
tom, and right in alternate code segments and you can waste an-
other 2.2K. Do this with several hundred unnecessary memory
variables and you will most certainly get to see "Out of
Memory Continue [Y/N/Well Maybe, But I Don't Think So]".
The second technique involves re-using symbol names as often
as possible within a given object file. You could use var_1,
var_2, ..... var_n in every code segment to minimize the number
of unique symbol names in each of your object files. A second
aspect of this technique is to compile as many of your code
segments as feasible into individual object files.
Most popular Linkers available for Clipper applications have
made inroads into solving the Symbol Table problem through tech-
niques generically referred to as Symbol Table Compaction -=-
much too diverse a subject to cover at this time (read the docu-
mentation on WarpLink among others for information on this tech-
nique). Some including Clipper 5.0(1)/RTLink address the problem
by pre-processing the Symbol Table during compile time -=- con-
verting memory variables into array elements (one array contain-
ing as many as 4092 memory variables only requires one 22 byte
space in the Symbol Table). There is a great deal to be said for
each of these techniques. Most, if not all, appear to be quite
effective.
ClipCASE is unique among third party Clipper applications
development platforms in that it addresses the Symbol Table prob-
lem at its source rather than dealing with its symptoms. Clip-
CASE was designed to provide Clipper Developers with a "ready-
made" front-end or System Shell as well as assisting throughout
the development and maintenance life-cycle of an application.
Products designed to help solve one of the major problems
facing developers, "development time", should definitely not be
guilty of contributing to the second most often mentioned prob-
lem, "memory usage". For this reason ClipCASE incorporates a
Symbol Table management technique called Source Level Symbol Ta-
ble Compression. This technique is virtually the same as that
used by pre-processors in that memory variables/symbols are con-
verted to array elements. The major difference is that ClipCASE
does it at the source code level prior to compiling rather than
between compile and link times. In ClipCASE this involved reduc-
ing more than 1,700 unique symbols to 56 arrays. The following
tables delineate the results of this approach.
Note: The graphics were trimmed to fit in sixty-five columns
rather than the eighty columns ClipCASE originally drew them in;
therefore the scale is not to correct proportion.
Before Source Level Symbol Table Compression ClipCASE's analysis
of its own Memory Map provided the following results:
Clip CASE OVERLAY ARCHITECTURE/MEMORY MAP ANALYSIS
±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±²²ÛÛÛ±±²²²²²²ÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛ
±±±±±±±±±±±±± Root ±±±±±±±±±±±±±1AÛ1B1C²²1D²²ÛÛÛ Symbol Table ÛÛÛ
±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±²²ÛÛÛ±±²²²²²²ÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛ
Overlay Area Size B/L Overlay Area Size B/L
Root 178.6 Kb 178.6 Kb Overlay # 1C 9.2 Kb 9.2 Kb
Overlay # 1A 7.3 Kb 7.3 Kb Overlay # 1D 23.4 Kb 23.4 Kb
Overlay # 1B 12.4 Kb 12.4 Kb Symbol Table 91.3 Kb 91.3 Kb
Total Load Module Size: 322.1 Kb -=|=- Baseline: 322.1 Kb
After Source Level Symbol Table Compression ClipCASE's report on
its Memory Map is:
Clip CASE OVERLAY ARCHITECTURE/MEMORY MAP ANALYSIS
±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±²²²ÛÛÛÛ±±±²²²²²²ÛÛÛÛÛÛÛÛÛÛÛÛÛ
±±±±±±±±±±±±±± Root ±±±±±±±±±±±±±±±±²1AÛ1BÛ±1C²²1D²²Û Symbl Tbl Û
±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±²²²ÛÛÛÛ±±±²²²²²²ÛÛÛÛÛÛÛÛÛÛÛÛÛ
Overlay Area Size B/L Overlay Area Size B/L
Root 180.0 Kb 178.6 Kb Overlay # 1C 11.6 Kb 8.6 Kb
Overlay # 1A 9.2 Kb 7.3 Kb Overlay # 1D 21.0 Kb 18.4 Kb
Overlay # 1B 14.0 Kb 12.4 Kb Symbol Table 53.4 Kb 91.3 Kb
Total Load Module Size: 289.2 Kb -=|=- Baseline: 322.1 Kb
By converting most (exceptions explained below) of its own
memory variables to arrays, ClipCASE reduced its own Symbol Table
from 91.3K to 53.4K or a reduction of 37.9K -=- 42%. This data
is based on 26,808 lines of code before compression and 26,432
lines of code after compression. The reduction in the number of
lines of code was due primarily to shorter array element names
making less need for wrapped code lines (wrapping done for read-
ability) and [blush] some lines of trash that were found during
debugging after compression. Both represent the same product
with absolutely no substantive change other than Symbol Table
Compression.
Symbols that cannot be compressed at the source level in-
clude parameter passing variables (receiving end), counter vari-
ables in FOR ... NEXT loops, and variables used to receive the
results of COUNT TO commands. Of course, function and procedure
names, including Clipper's functions such as MEMOEDIT(), STUFF(),
etc., continue to require a place in the Symbol Table.
Developers using ClipCASE as their applications development
platform may use ClipCASE to compress their own App's Symbol Ta-
ble if they desire. This technique is not recommended for the
faint hearted. ClipCASE accomplishes 80-85% of the compression
automatically but there is still a significant amount of editing
and debugging necessary after ClipCASE finishes the drudge work.
The real advantage to developers is that when they use ClipCASE's
System Shell for their application's front end, they do so with
essentially no inherited Symbol Table load. After source level
symbol table compression, editing and modifying the code becomes
something of an adventure. One does, however, adjust.
To accommodate the problem of being faced with a myriad of
array element numbers, rather than slightly more descriptive mem-
ory variable names, ClipCASE produces a script file for compiling
with a Norton's Guide Engine. The developer is then able to run
the Norton's Guide for the specific application being developed
or maintained behind his or her editor. This technique provides
the developer with context sensitive, hot-key access to the ap-
plication's complete Symbol Table (or the substituted array ele-
ments); Data Encyclopedia with system and descriptive names and
narrative explanations for all data bases and fields; data ele-
ment parameters, validation, and picture requirements; relational
integrity requirements; a complete system architecture or flow
diagram depicting internal relationships between all functions
and procedures; indexes including primary, secondary, and foreign
key fields; predefined filters or view criteria sets; and the
specifications and requirements that they are writing the code to
accommodate. The ability to access all of this information from
within their editor greatly simplifies the development process,
reduces development time, and helps to further insure that what
the developer is building is precisely what the client wants them
to build.
A somewhat, though not totally, unexpected fringe benefit of
ClipCASE doing Source Level Symbol Table Compression on itself
was a significant decrease in processing time. The tables below
were taken from screen captures of two of ClipCASE's source code
analysis routines and of packing two of ClipCASE's larger Data
Bases. The net results are, after compression ClipCASE operates
from 10 to 15% faster on average than it did before compression.
All processing from which this data was derived was done on the
same machine (AST Premium 386C/25MHz) -=- the hard disk was opti-
mized before each set of data was collected. The Estimated Time
to complete these processes was zeroed-out in order to clearly
show the time over estimates.
Before Compression:
ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍ͵ Clip CASE Source Code Analysis ÆÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»
º Cataloging Clip CASE Source Code Files Tot No Files: 55 º
º Processing: Screen Saver Record # 16 º
ÇÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄĶ
º Program/Procedure/Function Name Ln Cnt Ln No Action º
ÇÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄĶ
º System User's Manual Report 684 684 Analysis Compº
º Next Action Recommended/Required 903 903 Analysis Compº
º Index Criteria Definition 612 612 Analysis Compº
º Look-Up Tables & Pick-Lists 428 428 Analysis Compº
º System User Preferences 509 509 Analysis Compº
º System Profile 215 215 Analysis Compº
º Data Encyclopedia - System Data Bases 422 422 Analysis Compº
º Reports Management & Generation 991 991 Analysis Compº
º Screen Saver 145 145 Analysis Compº
ÇÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄĶ
º Totals: Files = 55 Line Cnt = 26,808 Code Cnt = 23,024 º
ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ͵ Escape To Abort ÆÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍͼ
Processing Source Code File For: Screen Saver
Est Time To Comp: 0 Min 0 Sec -=- Time Over Est: 46 Min 27 Sec
After Compression:
ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍ͵ Clip CASE Source Code Analysis ÆÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»
º Cataloging Clip CASE Source Code Files Tot No Files: 55 º
º Processing: Screen Saver Record # 16 º
ÇÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄĶ
º Program/Procedure/Function Name Ln Cnt Ln No Action º
ÇÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄĶ
º System User's Manual Report 684 684 Analysis Compº
º Next Action Recommended/Required 903 903 Analysis Compº
º Index Criteria Definition 612 612 Analysis Compº
º Look-Up Tables & Pick-Lists 428 428 Analysis Compº
º System User Preferences 509 509 Analysis Compº
º System Profile 215 215 Analysis Compº
º Data Encyclopedia - System Data Bases 422 422 Analysis Compº
º Reports Management & Generation 991 991 Analysis Compº
º Screen Saver 145 145 Analysis Compº
ÇÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄĶ
º Totals: Files = 55 Line Cnt = 26,432 Code Cnt = 21,205 º
ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ͵ Escape To Abort ÆÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍͼ
Processing Source Code File For: Screen Saver
Est Time To Comp: 0 Min 0 Sec -=- Time Over Est: 42 Min 29 Sec
Net Decrease in Processing Time: 3 Min 53 Sec -=- 8.54%
Before Compression:
ÉÍÍÍÍÍ͵ Clip CASE System Cross References & Flow Analysis ÆÍÍÍÍ»
Est Time To Comp: 0 Min 0 Sec -=- Time Over Est: 1 Min 20 Sec
After Compression:
ÉÍÍÍÍÍ͵ Clip CASE System Cross References & Flow Analysis ÆÍÍÍÍ»
Est Time To Comp: 0 Min 0 Sec -=- Time Over Est: 1 Min 12 Sec
Net Decrease in Processing Time: 8 Seconds -=- 10%
Before Compression:
ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»
º Packing Data Base! -- No Esc At This Time º
º Data Base File Name: SYS_PRGS.DBF º
º Packing Will Require Approximately: 00:01:14 º
ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍͼ
After Compression:
ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»
º Packing Data Base! -- No Esc At This Time º
º Data Base File Name: SYS_PRGS.DBF º
º Packing Will Require Approximately: 00:01:00 º
ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍͼ
Net Decrease In Processing Time: 14 Seconds -=- 19%
Before Compression:
ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»
º Packing Data Base! -- No Esc At This Time º
º Data Base File Name: SYS_DDIR.DBF º
º Packing Will Require Approximately: 00:00:54 º
ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍͼ
After Compression:
ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»
º Packing Data Base! -- No Esc At This Time º
º Data Base File Name: SYS_DDIR.DBF º
º Packing Will Require Approximately: 00:00:46 º
ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍͼ
Net Decrease In Processing Time: 8 Seconds -=- 15%
In the examples above comparing processing times for packing
two different Data Bases it was, at first, assumed that the ef-
fective decrease in processing time was due to more Free Pool
Memory being available. However, when Free Pool was reduced to
the pre-compression value using Tom Rettig's MEMHOG.COM function
there was no increase in processing time.
In summary, ClipCASE provides a uniquely different approach
to dealing with memory usage problems. Whether the developer
elects to compress the symbol table for his or her own applica-
tion is secondary to the fact that ClipCASE provides an extensive
library of data management functions and features with virtually
no load added to the application's symbol table and improves exe-
cution speed by as much as 10-15% for all of those library func-
tions. The ability to automatically create Norton's Guides "on-
the-fly" for applications while they are still being developed
should prove invaluable to developers under any circumstance.
by
Riley Paulk
The Author
of
ClipCASE
Out Of Memory Continue [Y/N]
If you are a Clipper Developer and you have written anything
approaching a medium to large application in S'87 then you have
probably had occasion to see the message above. One is espe-
cially intrigued by Nantucket's recovery option "Continue [Y/N]"
-=- what they should have said is "Continue [Y/N/Well Maybe, But
I don't Think So]". Memory management problems have been a thorn
in the side of Clipper Developers for as long as they have been
writing Clipper Applications. Roger Donay of dClip fame says in
an article he wrote on Clipper Linkers "If you ask the Clipper
programmer about his biggest frustrations, he/she will virtually
always answer the same way - memory usage and development time."
This writer considers memory management problems to be something
like an albatross hanging around his neck. The following infor-
mation is provided in the hope that it will help others through
the process of solving these problems.
This information should be of use to you if:
1. You are still writing applications in Clipper S'87
Code.
2. You are maintaining applications that were written in
Clipper S'87 Code.
3. You have S'87 applications that you are considering
investing the time necessary to port them over to
Clipper 5.0(1).
4. You have applications in which you had to use Nan-
tucket's SWITCH or any number of third-party SWAP rou-
tines for clearing up enough RAM to allow your applica-
tion to run in that little 640K box we all have to live
in.
5. You use compilers such as WarpLink, Blinker, dClip with
a myriad of pre-processors, dynamic overlays, and other
magic aimed at dealing with memory management problems.
6. And, maybe, even if you have already made the quantum
leap to Clipper 5.0(1), are writing exclusively in
5.0(1) and have nothing to do with S'87.
The number one perpetrator of memory management problems in
any Clipper application is the Symbol Table. With well managed
overlays whether static, as with executables produced using
Plink86 Plus, or dynamic overlays produced with dClip, WarpLink,
Blinker, or RTLink 5.0 a tremendous amount of code can success-
fully be incorporated into a Clipper application. The problem is
that no matter how well the overlays are managed the Symbol Table
keeps growing.
A topic so dear to the heart of Clipper programmers as this
has already been addressed extensively by numerous authorities in
the field. The documentation that accompanies Steve Steiner's
excellent memory management tools library called SmartMem and a
text file available on many Clipper oriented BBS's called CLI-
PLINK.ZIP written by Roger Donay of dClip are both especially
informative on the subject of Symbol Tables and the management of
problems related to Symbol Tables. Rather than covering the same
material again in this article, it is recommended that, if you
are not already familiar with Clipper's use of Symbol Tables, you
read those documents or other sources of similar information.
In brief, memory variable references as well as references
to procedures and functions, including those in Clipper itself
and its extend library, are held in the root overlay area [that
portion of the executable that stays in RAM throughout execution]
in an area referred to as the Symbol Table. Essentially each
uniquely named "symbol" in a logical overlay area requires 22
bytes of RAM. Therefore, as your application grows, so does your
Symbol Table. The code for your application can be split into
manageable size parcels either through static overlays or the use
of dynamic overlay techniques but the Symbol Table must remain
intact and must, in most cases, remain in the root area. Unless
corrective action is taken, this Symbol Table will eventually
become the limiting factor on the size -=- read that power -=- of
your applications.
There are many techniques available for coping with the size
of the Symbol Table. The first, and obviously most effective,
technique is don't create the symbol to begin with. Use con-
stants instead of memory variables. A simple example of this
technique is:
t = 10
l = 20
b = 16
r = 59
@ t, l, b, r BOX "ÉÍ»º¼ÍȺ "
Which is a perfectly ridiculous way to accomplish:
@ 10, 20, 16, 59 BOX "ÉÍ»º¼ÍȺ "
If you use the first method to draw a box in each of fifty
code segments compiled into twenty-five object files with two
code segments in each object file you will have wasted 2.2K of
RAM. Remember that it is uniquely named symbols within an object
file that counts. Change the variable names to top, left, bot-
tom, and right in alternate code segments and you can waste an-
other 2.2K. Do this with several hundred unnecessary memory
variables and you will most certainly get to see "Out of
Memory Continue [Y/N/Well Maybe, But I Don't Think So]".
The second technique involves re-using symbol names as often
as possible within a given object file. You could use var_1,
var_2, ..... var_n in every code segment to minimize the number
of unique symbol names in each of your object files. A second
aspect of this technique is to compile as many of your code
segments as feasible into individual object files.
Most popular Linkers available for Clipper applications have
made inroads into solving the Symbol Table problem through tech-
niques generically referred to as Symbol Table Compaction -=-
much too diverse a subject to cover at this time (read the docu-
mentation on WarpLink among others for information on this tech-
nique). Some including Clipper 5.0(1)/RTLink address the problem
by pre-processing the Symbol Table during compile time -=- con-
verting memory variables into array elements (one array contain-
ing as many as 4092 memory variables only requires one 22 byte
space in the Symbol Table). There is a great deal to be said for
each of these techniques. Most, if not all, appear to be quite
effective.
ClipCASE is unique among third party Clipper applications
development platforms in that it addresses the Symbol Table prob-
lem at its source rather than dealing with its symptoms. Clip-
CASE was designed to provide Clipper Developers with a "ready-
made" front-end or System Shell as well as assisting throughout
the development and maintenance life-cycle of an application.
Products designed to help solve one of the major problems
facing developers, "development time", should definitely not be
guilty of contributing to the second most often mentioned prob-
lem, "memory usage". For this reason ClipCASE incorporates a
Symbol Table management technique called Source Level Symbol Ta-
ble Compression. This technique is virtually the same as that
used by pre-processors in that memory variables/symbols are con-
verted to array elements. The major difference is that ClipCASE
does it at the source code level prior to compiling rather than
between compile and link times. In ClipCASE this involved reduc-
ing more than 1,700 unique symbols to 56 arrays. The following
tables delineate the results of this approach.
Note: The graphics were trimmed to fit in sixty-five columns
rather than the eighty columns ClipCASE originally drew them in;
therefore the scale is not to correct proportion.
Before Source Level Symbol Table Compression ClipCASE's analysis
of its own Memory Map provided the following results:
Clip CASE OVERLAY ARCHITECTURE/MEMORY MAP ANALYSIS
±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±²²ÛÛÛ±±²²²²²²ÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛ
±±±±±±±±±±±±± Root ±±±±±±±±±±±±±1AÛ1B1C²²1D²²ÛÛÛ Symbol Table ÛÛÛ
±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±²²ÛÛÛ±±²²²²²²ÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛ
Overlay Area Size B/L Overlay Area Size B/L
Root 178.6 Kb 178.6 Kb Overlay # 1C 9.2 Kb 9.2 Kb
Overlay # 1A 7.3 Kb 7.3 Kb Overlay # 1D 23.4 Kb 23.4 Kb
Overlay # 1B 12.4 Kb 12.4 Kb Symbol Table 91.3 Kb 91.3 Kb
Total Load Module Size: 322.1 Kb -=|=- Baseline: 322.1 Kb
After Source Level Symbol Table Compression ClipCASE's report on
its Memory Map is:
Clip CASE OVERLAY ARCHITECTURE/MEMORY MAP ANALYSIS
±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±²²²ÛÛÛÛ±±±²²²²²²ÛÛÛÛÛÛÛÛÛÛÛÛÛ
±±±±±±±±±±±±±± Root ±±±±±±±±±±±±±±±±²1AÛ1BÛ±1C²²1D²²Û Symbl Tbl Û
±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±²²²ÛÛÛÛ±±±²²²²²²ÛÛÛÛÛÛÛÛÛÛÛÛÛ
Overlay Area Size B/L Overlay Area Size B/L
Root 180.0 Kb 178.6 Kb Overlay # 1C 11.6 Kb 8.6 Kb
Overlay # 1A 9.2 Kb 7.3 Kb Overlay # 1D 21.0 Kb 18.4 Kb
Overlay # 1B 14.0 Kb 12.4 Kb Symbol Table 53.4 Kb 91.3 Kb
Total Load Module Size: 289.2 Kb -=|=- Baseline: 322.1 Kb
By converting most (exceptions explained below) of its own
memory variables to arrays, ClipCASE reduced its own Symbol Table
from 91.3K to 53.4K or a reduction of 37.9K -=- 42%. This data
is based on 26,808 lines of code before compression and 26,432
lines of code after compression. The reduction in the number of
lines of code was due primarily to shorter array element names
making less need for wrapped code lines (wrapping done for read-
ability) and [blush] some lines of trash that were found during
debugging after compression. Both represent the same product
with absolutely no substantive change other than Symbol Table
Compression.
Symbols that cannot be compressed at the source level in-
clude parameter passing variables (receiving end), counter vari-
ables in FOR ... NEXT loops, and variables used to receive the
results of COUNT TO commands. Of course, function and procedure
names, including Clipper's functions such as MEMOEDIT(), STUFF(),
etc., continue to require a place in the Symbol Table.
Developers using ClipCASE as their applications development
platform may use ClipCASE to compress their own App's Symbol Ta-
ble if they desire. This technique is not recommended for the
faint hearted. ClipCASE accomplishes 80-85% of the compression
automatically but there is still a significant amount of editing
and debugging necessary after ClipCASE finishes the drudge work.
The real advantage to developers is that when they use ClipCASE's
System Shell for their application's front end, they do so with
essentially no inherited Symbol Table load. After source level
symbol table compression, editing and modifying the code becomes
something of an adventure. One does, however, adjust.
To accommodate the problem of being faced with a myriad of
array element numbers, rather than slightly more descriptive mem-
ory variable names, ClipCASE produces a script file for compiling
with a Norton's Guide Engine. The developer is then able to run
the Norton's Guide for the specific application being developed
or maintained behind his or her editor. This technique provides
the developer with context sensitive, hot-key access to the ap-
plication's complete Symbol Table (or the substituted array ele-
ments); Data Encyclopedia with system and descriptive names and
narrative explanations for all data bases and fields; data ele-
ment parameters, validation, and picture requirements; relational
integrity requirements; a complete system architecture or flow
diagram depicting internal relationships between all functions
and procedures; indexes including primary, secondary, and foreign
key fields; predefined filters or view criteria sets; and the
specifications and requirements that they are writing the code to
accommodate. The ability to access all of this information from
within their editor greatly simplifies the development process,
reduces development time, and helps to further insure that what
the developer is building is precisely what the client wants them
to build.
A somewhat, though not totally, unexpected fringe benefit of
ClipCASE doing Source Level Symbol Table Compression on itself
was a significant decrease in processing time. The tables below
were taken from screen captures of two of ClipCASE's source code
analysis routines and of packing two of ClipCASE's larger Data
Bases. The net results are, after compression ClipCASE operates
from 10 to 15% faster on average than it did before compression.
All processing from which this data was derived was done on the
same machine (AST Premium 386C/25MHz) -=- the hard disk was opti-
mized before each set of data was collected. The Estimated Time
to complete these processes was zeroed-out in order to clearly
show the time over estimates.
Before Compression:
ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍ͵ Clip CASE Source Code Analysis ÆÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»
º Cataloging Clip CASE Source Code Files Tot No Files: 55 º
º Processing: Screen Saver Record # 16 º
ÇÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄĶ
º Program/Procedure/Function Name Ln Cnt Ln No Action º
ÇÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄĶ
º System User's Manual Report 684 684 Analysis Compº
º Next Action Recommended/Required 903 903 Analysis Compº
º Index Criteria Definition 612 612 Analysis Compº
º Look-Up Tables & Pick-Lists 428 428 Analysis Compº
º System User Preferences 509 509 Analysis Compº
º System Profile 215 215 Analysis Compº
º Data Encyclopedia - System Data Bases 422 422 Analysis Compº
º Reports Management & Generation 991 991 Analysis Compº
º Screen Saver 145 145 Analysis Compº
ÇÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄĶ
º Totals: Files = 55 Line Cnt = 26,808 Code Cnt = 23,024 º
ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ͵ Escape To Abort ÆÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍͼ
Processing Source Code File For: Screen Saver
Est Time To Comp: 0 Min 0 Sec -=- Time Over Est: 46 Min 27 Sec
After Compression:
ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍ͵ Clip CASE Source Code Analysis ÆÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»
º Cataloging Clip CASE Source Code Files Tot No Files: 55 º
º Processing: Screen Saver Record # 16 º
ÇÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄĶ
º Program/Procedure/Function Name Ln Cnt Ln No Action º
ÇÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄĶ
º System User's Manual Report 684 684 Analysis Compº
º Next Action Recommended/Required 903 903 Analysis Compº
º Index Criteria Definition 612 612 Analysis Compº
º Look-Up Tables & Pick-Lists 428 428 Analysis Compº
º System User Preferences 509 509 Analysis Compº
º System Profile 215 215 Analysis Compº
º Data Encyclopedia - System Data Bases 422 422 Analysis Compº
º Reports Management & Generation 991 991 Analysis Compº
º Screen Saver 145 145 Analysis Compº
ÇÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄĶ
º Totals: Files = 55 Line Cnt = 26,432 Code Cnt = 21,205 º
ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ͵ Escape To Abort ÆÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍͼ
Processing Source Code File For: Screen Saver
Est Time To Comp: 0 Min 0 Sec -=- Time Over Est: 42 Min 29 Sec
Net Decrease in Processing Time: 3 Min 53 Sec -=- 8.54%
Before Compression:
ÉÍÍÍÍÍ͵ Clip CASE System Cross References & Flow Analysis ÆÍÍÍÍ»
Est Time To Comp: 0 Min 0 Sec -=- Time Over Est: 1 Min 20 Sec
After Compression:
ÉÍÍÍÍÍ͵ Clip CASE System Cross References & Flow Analysis ÆÍÍÍÍ»
Est Time To Comp: 0 Min 0 Sec -=- Time Over Est: 1 Min 12 Sec
Net Decrease in Processing Time: 8 Seconds -=- 10%
Before Compression:
ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»
º Packing Data Base! -- No Esc At This Time º
º Data Base File Name: SYS_PRGS.DBF º
º Packing Will Require Approximately: 00:01:14 º
ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍͼ
After Compression:
ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»
º Packing Data Base! -- No Esc At This Time º
º Data Base File Name: SYS_PRGS.DBF º
º Packing Will Require Approximately: 00:01:00 º
ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍͼ
Net Decrease In Processing Time: 14 Seconds -=- 19%
Before Compression:
ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»
º Packing Data Base! -- No Esc At This Time º
º Data Base File Name: SYS_DDIR.DBF º
º Packing Will Require Approximately: 00:00:54 º
ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍͼ
After Compression:
ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»
º Packing Data Base! -- No Esc At This Time º
º Data Base File Name: SYS_DDIR.DBF º
º Packing Will Require Approximately: 00:00:46 º
ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍͼ
Net Decrease In Processing Time: 8 Seconds -=- 15%
In the examples above comparing processing times for packing
two different Data Bases it was, at first, assumed that the ef-
fective decrease in processing time was due to more Free Pool
Memory being available. However, when Free Pool was reduced to
the pre-compression value using Tom Rettig's MEMHOG.COM function
there was no increase in processing time.
In summary, ClipCASE provides a uniquely different approach
to dealing with memory usage problems. Whether the developer
elects to compress the symbol table for his or her own applica-
tion is secondary to the fact that ClipCASE provides an extensive
library of data management functions and features with virtually
no load added to the application's symbol table and improves exe-
cution speed by as much as 10-15% for all of those library func-
tions. The ability to automatically create Norton's Guides "on-
the-fly" for applications while they are still being developed
should prove invaluable to developers under any circumstance.
Comments
Post a Comment