Reprinted from MUG QUARTERLY, Vol. XIII, Number 3/4. The MUG Quarterly has been replaced by M Computing which is published by:

M Technology Association
1738 Elton Road, Suite 205
Silver Spring, MD 20903 USA
Phone 301-431-4070
Fax 301-431-0017
Email mta@mtechnology.org
Website www.mtechnology.org

The OCR process may have introduced errors. Other than a spell check, the contents of the original document has not been changed, although the document has been formatted.

System Performance: A Benchmark Study on MUMPS and Other Systems

Casimir M. Alonzo
Centro de Calculo de Sabadell.
S.A. Barcelona, Spain

Introduction

Very often when an organization buys a minicomputer to solve its administrative problems, it discovers that, once installed, the computer seems too slow and too small to be able to handle the work for which it was installed. One would expect this to always occur eventually, as the organization adjusts to the computer and the workload expands. However, often it occurs immediately, perhaps because of under buying on the part of the purchaser, or because vendor-purchaser computer is poor. The purchase may be based on measures of computer performance such as cycles/second which tell little about the system performance in the operating environment. When this happens, the organization is forced to, either:

Is it possible to avoid this situation? We have seen a continued evolution in the hardware area, with the popularization of mini, micro and super minicomputers, while the basic software used tends to be a replication of the same old tools originally designed for use on multi million dollar mainframes. These commonly used languages and database systems, because of their basic concepts, tend to be both highly inefficient in terms of machine resources consumed and hardly adapted to interactive use. The margin for error is smaller on these small computers than on mainframes, and therefore some of the traditional language approaches fail. All these issues lead to the problems stated above.

Fortunately, technology has also evolved in the software field, and now we have available powerful tools like MUMPS, an ANSI standard language that was specifically designed to be run on mini, micro and super minicomputers for interactive database applications. Because we all normally work with MUMPS, we tend to under evaluate the performance improvements obtained when compared with traditional systems. MUMPS is not only a programming environment which improves programmers' productivity, but it also offers significantly improved performance in system throughput and disk utilization.

The Benchmark Study

In order to avoid purchasing a computer system too small for the job, a department store in Spain recently required bidders to participate in a unique benchmark study. The benchmark used took into consideration the entire environment in which the computer system would be functioning. Systems which were compared included one which utilized MUMPS.

Thus, we had the opportunity to compare MUMPS with other languages and database systems in terms of disk occupation, processing speed, and lines of code required. We found MUMPS to perform incredibly well: fewer lines of code were needed to do a given task, MUMPS outperformed machines which were theoretically superior in speed and used far less disk space than the competitors. This can be explained by the advanced internal design characteristics of MUMPS and by its suitability to interactive database applications.

As a mandatory part of the proposals requested by a small department store considering the computerization of its Administration Department, the benchmark described below was required. The department store has 600 employees with revenues of $130,000,000 per year, and is one of the larger department stores in Spain.

The department store prepared an RFP which clearly indicated that this benchmark study would be done. The vendor was to provide the performance measures required. The benchmarks were to be run on the equipment to be supplied, as a single job on a single terminal. The vendors were allowed to use as many programmers as they wanted to produce the software.

To be sure the measures reported were honest, vendors were required to supply their source code and to explain how it worked. The source code was to be examined by the purchaser's programmer, and the bidder had to compile the code in front of the purchaser. The initial database was to be shown to be empty, then it was created. The code had to be run in the presence of the purchaser. Finally, the purchaser could ask the bidder to use different data: software to allow such modification was part of the requirements.

Data was created by generating random numbers; the time for database creation included the time to generate random data and the time to load this from a sequential file, if such a file was used.

The purchaser kept all bidders informed about the results obtained by competitors, thus forcing every vendor to obtain the best possible performance from their system.

The benchmark consisted of:

(1) Creation of a database with the following structure:

• Item file (1,000 records), containing an item number for every item in the store.
• Color file (6,000 records) containing 6 "color" records per "item".
• Size file (48,000 records) containing 8 "size" records per "item-color".
• Store file (288,000 records) containing 6 "store" records per "item-color-size". An item might be found at any of these stores.
• Supplier (manufacturer) file (20 records).
• Category file (50 records -- groups of items, e.g. men's clothing).

Purchase prices and selling prices could be present at any level.

Figure 3: Performance measurements obtained with 32 bit computers.

Now, if we look at the 16 bit benchmark results, as presented in Figure 4, they arc very similar to the 32 bit results. (The unexpectedly low database occupation of the Reality 2000 system was due to the use of single digit fields, a technique which would not have stood up to close scrutiny.) Moreover, a 16 bit minicomputer running MUMPS outperformed 32 bit computers running conventional software. It is also interesting to note that not one modification of the source code was needed in order to move the MUMPS programs between the 16 and the 32 bit computers.

Figure 4: Performance measurements obtained with 16 bit computers.

MUMPS versus COBOL

In order to see what portion of the difference in performance was due to the computer language used, we made our own comparison of MUMPS and COBOL running the benchmark on the same computer hardware. MUMPS programs for the benchmarks were written by a single systems programmer who had not been programming MUMPS COBOL programs were v written by a skilled COBOL programmer from our company with 6 years full-time commercial experience writing COBOL software. The machine selected was a 32 bit minicomputer (VAX 750) running a vendor supplied COBOL compiler and ISAM file system, against InterSystems' M/VX MUMPS language and database system. The results obtained are shown in Figure 5. This chart shows the impressive performance of the MUMPS M/VX software ware over the vendor's COBOL.

Outcome of Bidding

The contract was won by Calculo de Sabadell. The full configuration which was installed consisted of a DEC VAX-11/750 with 1 megabyte of memory, a TU-80 tape drive, an RA-81 456 megabyte disk drive, and 16 terminals. Applications planned include payroll, general ledger, accounts receivable, accounts payable, purchasing, stock control, and sales analysis. Expansion to up to 40 terminals is planned. It is likely that two more VAX systems will be installed at different stores in the future. Also, point-of-sales information will be acquired.

*COBOL measurement divided by MUMPS measurement

Figure 5: Comparison of MUMPS and COBOL running the benchmarks on the same computer hardware

Figure 1: Technical Characteristics of the 32 bit computers which participated in the benchmarking.

(b) Relations: 

• Item-color-size-store (343,000 relations); for example, where is each item of each color and size located.
• Store-item-color-size (343,000 relations); for example, what items of which colors and sizes are located in this store.
• Supplier-item (1,000 relations); for example, what items does this supplier provide.
• Category-item (1,000 relations); for example, what items do we sell in this category.

(2) Use of a Database Query Program giving a stock and sales chart as well as a stock value per category item-color-size-store. The query required the use of the "category-item" and the "'item-color-size-store" relations. Two different queries were performed: a "long query" which totaled 50 items in one category located in all stores and including all colors and sizes. This query required access to 17,650 records; a "short query" which totaled only one item of one color across stores and sizes. This required access to only 50 records.

The benchmark testing involved comparisons between several different languages and between several 16 bit and 32 bit computers (Figure 1 and 2 for technical characteristics). After examining the multi vendor benchmark where the maximum possible performance was obtained from every system, we will describe an additional benchmark performed to determine the effect of programming language independent of computer hardware.

Over these eight computers, the MUMPS systems were not run on, a priori, the faster computers, at least according to the vendors' announced theoretical processor performance.

Let us first analyze the performance obtained with 32 bit computer as shown in Figure 3. The MUMPS system showed much better performance than its competitors on almost every test run. On the Database Creation test, MUMPS was from 30 to 7 times faster than the conventional systems. The MUMPS database needed only between 0.5 and 0.25 of the disk space used by the competitors' databases. And, if we look at the difficulty of programming, we again see that MUMPS SIP requires less than one third the lines of code of the others. So, on 32 bit computers MUMPS was only beaten once, while it was clearly superior to its competitors in overall system performance.

Figure 2: Technical characteristics of the 16 bit computers which participated in the benchmarking.

The results found on the above benchmarks show that, by choosing MUMPS software, one can obtain up to five time more power from a computer, thus overcoming the typical performance problems on minicomputers running interactive database applications.

Some criticism of MUMPS has been based on the feet that its interpreter approach may impose severe penalties on performance of MUMPS applications. The benchmark results shown here clearly state that MUMPS not only is not slower than other software, but that MUMPS offers an incredible performance Increase over conventional systems.

A Thought for MUMPS Users

Finally, I would like to stress one point. Because we all normally work with MUMPS, we tend to undervalue the performance improvements obtained when compared with traditional systems. MUMPS is not only an incredible tool for increasing analyst and programmer productivity, but MUMPS offers an incredible performance in system throughput and disk occupation. This point has to be made public to the general computer community. We have to commit ourselves to this large but important task, submitting papers to the most commonly read computer magazines, doing presentations at "Computing Conferences," etc. If, we do so, we will increase the world's interest in MUMPS, and thus widen the MUMPS community.

Chris Bonnici
If you have any other document we could include here, send an e-mail to chribonn@softhome.net.

Last Updated: 29 September 1997

T^{he Index} or ^{Go U}P