gcc, egcs and pgcc comparisson

GCC, EGCS and PGCC comparisson

New I'm arranging some new data, it includes results from 26 different compilations tested on a K6-2 300 MHz and a Pentium MMX 233 MHz. I'll need some weeks before I can replace the old values by the new ones, but I also included an analisys of the assembler generated by gcc compared with the one generated by MSVC that is very interesting.
I think anybody involved in the gcc optimization should read it.

The "Linux Game Development Center" people did a re-arrangement of the data presented here taking MSVC compiler as reference, you can find it in: http://sunsite.auc.dk/linuxgames/articles/tropea_compilers.html .

At first I must say this comparisson isn't very accurate but is just yet another value to take in count.
I used the BYTE benchmarks, the C sources says: "BYTEmark (tm) BYTE MAGAZINE'S NATIVE MODE BENCHMARKS FOR CPU/FPU ver 2.0 Rick Grehan, BYTE Magazine". They are supposed to be portable and report values with some meaning, not just numbers ;-).
The benchmarks are just some very well known algorithms implemented by the BYTE people. I think they are a little bit adjusted for Watcom compilers but they are ok.
I used an AMD K6 266 MHz computer so the values in an Intel CPU could be different but I think they will be very similar.
All the test were performed under Wincrap 95 using djgpp compiler.
The tables are very wide, I don't know a simple way to solve it.

Obtained values

Operation\Compiler	gcc 2.8.0 No optimizations (3)	gcc 2.8.0 -O2 (3)	gcc 2.8.0 -O3 (1)(3)	gcc 2.8.0 -O3 (1)(2)(3)	egcs 1.1a -O3 (1)(4)	egcs 1.1a -O3 (1)(2)(4)	pgcc 2.90.23 (egcs 1.0.1) -O3 (1)(4)	pgcc 2.90.23 (egcs 1.0.1) -O6 (1)(4)(5)	egcs 1.1.1 -O2 (2)(6)	egcs 1.1.1 -O3 (1)(2)(6)	egcs 1.1.1 -O3 (1)(6)	Visual C++ 5.0	Watcom 10.0a
NUMERIC SORT:	50.158649	138.177184	138.810736	142.027848	139.839147	139.346933	137.813019	125.974885	139.437737	123.297358	125.473971	153.228628	125.781775
STRING SORT:	7.7376	9.251276	9.27043	9.309544	9.251276	9.212968	9.026465	9.194205	8.936741	9.157474	9.194596	16.248662	12.723533
BITFIELD:	11160540.08	26483635.13	26449374.67	26483924.69	25654847.51	27396247.33	22017659.47	22793551.47	22793335.53	27342117.75	27397278.61	35894514.24	24903701.78
FP EMULATION:	3.87234	6.678899	7.567698	7.663158	7.695767	7.982609	8.481553	8.896277	6.4	8.164486	7.982609	6.573861	6.069806
FOURIER:	2539.842356	2390.351929	2579.286012	2519.12912	2357.916522	2387.837282	2342.626316	2444.449031	2300.573279	2361.259221	2346.957285	2857.254474	2269.467655
ASSIGNMENT:	0.647699	1.292174	1.187013	1.052078	1.378827	1.105281	1.058195	1.253966	1.100869	1.092	1.338439	1.959249	0.803368
IDEA:	150.046946	198.290958	245.006726	206.823983	241.724272	201.030324	215.389633	272.289808	150.046946	204.172272	236.753824	256.719471	249.36021
HUFFMAN:	50.091743	109.861818	120.266667	123.451456	123.213592	122.033333	132.99348	130.005415	122.5	122.033333	119.73913	173.959603	168.270136
NEURAL NET:	0.456529	1.582609	1.6	1.565591	1.343906	1.381805	1.63964	1.427451	1.128119	1.327083	1.292121	1.295375	1.338978
LU DECOMPOSITION:	13.728705	30.777558	30.8	30.096342	34.4064	33.6	29.89	34.462341	35.549059	33.222749	34.339959	49.900198	40.634605
INTEGER INDEX:	1.990743	3.709147	3.897891	3.775678	3.983518	3.805239	3.763078	3.980056	3.429679	3.748886	3.931875	4.998914	3.842807
FLOATING-POINT INDEX:	1.216572	2.36151	2.431587	2.376735	2.310356	2.323255	2.350494	2.387054	2.185277	2.275059	2.275287	2.754111	2.408228

Note: Even when nobody asked, and I got more than 150 hits in less than 24 hs. and three comments, the values here are iteration/second. It means that greater is better.

Normalized values, the reference is the Watcom version

Note: Mark Potter teach me that arithmetic averages can generate wrong results and that some people use them to fool others so here I use a better methode.

Operation\Compiler	gcc 2.8.0 No optimizations (3)	gcc 2.8.0 -O2 (3)	gcc 2.8.0 -O3 (1)(3)	gcc 2.8.0 -O3 (1)(2)(3)	egcs 1.1a -O3 (1)(4)	egcs 1.1a -O3 (1)(2)(4)	pgcc 2.90.23 (egcs 1.0.1) -O3 (1)(4)	pgcc 2.90.23 (egcs 1.0.1) -O6 (1)(4)(5)	egcs 1.1.1 -O2 (2)(6)	egcs 1.1.1 -O3 (1)(2)(6)	egcs 1.1.1 -O3 (1)(6)	Visual C++ 5.0	Watcom 10.0a
NUMERIC SORT:	0.398775172	1.09854694	1.103583854	1.129160787	1.111760007	1.107846769	1.095651727	1.001535278	1.108568686	0.980248196	0.997552873	1.218210094	1
STRING SORT:	0.608132977	0.727099619	0.728605019	0.731679165	0.727099619	0.72408882	0.709430706	0.722614151	0.702378891	0.719727296	0.722644882	1.27705583	1
BITFIELD:	0.448147837	1.063441707	1.062065989	1.063453334	1.030162011	1.100087352	0.884111915	0.915267605	0.915258934	1.097913796	1.100128762	1.44133248	1
FP EMULATION:	0.637967671	1.100348018	1.246777574	1.262504601	1.26787693	1.315134125	1.397335104	1.465660846	1.054399432	1.345098344	1.315134125	1.083043017	1
FOURIER:	1.119135737	1.053265476	1.136515872	1.110008823	1.038973398	1.052157442	1.032236045	1.077102388	1.013706132	1.040446298	1.034144408	1.258997663	1
ASSIGNMENT:	0.806229524	1.608445943	1.477545782	1.309584151	1.716308093	1.375809094	1.317198345	1.560886169	1.370317215	1.359277442	1.666034744	2.438793928	1
IDEA:	0.601727701	0.795198873	0.982541385	0.829418547	0.969377881	0.806184451	0.863769055	1.091953716	0.601727701	0.818784489	0.949445078	1.029512571	1
HUFFMAN:	0.297686471	0.652889578	0.714723776	0.73365042	0.732236836	0.725222763	0.790357001	0.772599453	0.727996084	0.725222763	0.711588716	1.033811508	1
NEURAL NET:	0.340953324	1.181952952	1.194941216	1.169243259	1.003680419	1.031984842	1.224545885	1.066075021	0.842522431	0.991116359	0.9650054	0.967435611	1
LU DECOMPOSITION:	0.337857474	0.75742235	0.757974638	0.74065792	0.846726577	0.826881423	0.735579932	0.848103261	0.874846919	0.817597439	0.845091493	1.228022224	1
INTEGER INDEX:	0.518043972	0.965218134	1.014334313	0.982531259	1.036616723	0.990223813	0.979252406	1.035715819	0.89249317	0.975559272	1.023177849	1.300849613	1
FLOATING-POINT INDEX:	0.505173098	0.980600674	1.009699663	0.98692275	0.95935933	0.96471555	0.976026356	0.991207643	0.907421141	0.944702495	0.94479717	1.143625521	1
Product:	0.00129085	0.735960331	1.137220628	0.849714799	1.135718225	0.83816604	0.802872164	1.245023761	0.33702256	0.709021306	0.990083985	9.428515061	1
Geometric average:	0.514147308	0.969807282	1.012941753	0.983846439	1.012807852	0.982501011	0.9782833	1.022157369	0.896945174	0.966197543	0.999003946	1.251538834	1
Same in percent:	51.41%	96.98%	101.29%	98.38%	101.28%	98.25%	97.83%	102.22%	89.69%	96.62%	99.90%	125.15%	100.00%

Graphics:

All in one it contains all the data, is hard to use but you can see the weak and strong points of GNU compilers.
Integer and floating point indexes it contains the geometric averages of the integer and floating point test separated.
Overall it contains the geometric averages of all the tests.

References:

(1) -fomit-frame-pointer -ffast-math
(2) -mcpu=i586 -march=i586
(3) djgpp v2.01 and binutils 2.8.1
(4) djgpp v2.02 (981115) binutils 2.9.1
(5) -mcpu=amdk6 -march=amdk6
(6) djgpp v2.02 release and binutils 2.9.1

My conclusions:

Even when particular algorithms are benefited by some compiler the whole suite of benchmarks gives almost the same values when we add the partial results giving the same weight to each test.
The only compiler that gives better performance is the Microsoft VC++ 5.0. This compiler is really well fine tuned for Pentium processors. It doesn't surprise me because BYTE compared compilers from Symantec, Borland, Watcom (too versions), Intel and Microsoft in january of 1998. My results are the same. GCC is better than Borland 5.02 and Symantec 7.5 (the worst); is comparable to Watcom 10 and 11 and is worst than Intel 2.4 and Microsoft 5.0.
Is interesting the result of gcc without optimizations, it gets about half of the points.
Basically all the compilers performed OK and with almost no difference. The best for overall was the fine tunned pgcc using -O6 and specifying AMD K6 as target. But even in this case the result was less than 5% better than Watcom. Even comparing the best case with the worst (gcc -O2) the difference is under 5%.
The conclusion is that for non-specific uses all are almost the same. In fact gcc 2.8.0 using -O3 performs very well in Pentium processors.
I think the weak point of gcc is the use of registers as parameters. This task is done very well by Watcom's compiler and with GNU compilers is something the programmer must do by hand making it impossible to use.
BTW according to this benchmark the K6 266 is a 345.6 MHz Pentium for integers and a 216.72 MHz Pentium for FPU.

Visited since December 2, 1998:

Copyright © 1998,1999 by Salvador E. Tropea.
If you want to use these values for some publication just let me know.
HTML created with Setedit, my own text editor ;-). The macros, column operations, regular expressions search & replace, etc. helped a lot.