Introduction to Neural Network
By Hathaithep Wongsuvan
What is a neural network (NN)?
First of all, when we are talking about a neural network, we *should* usually
better say "artificial neural network" (ANN), because that is what we mean
most of the time. Biological neural networks are much more complicated in
their elementary structures than the mathematical models we use for ANNs.
A vague description is as follows:
An ANN is a network of many very simple processors ("units"), each possibly
having a (small amount of) local memory. The units are connected by
unidirectional communication channels ("connections"), which carry numeric
(as opposed to symbolic) data. The units operate only on their local data and on
the inputs they receive via the connections.
The design motivation is what distinguishes neural networks from other
mathematical techniques:
A neural network is a processing device, either an algorithm, or actual
hardware, whose design was motivated by the design and functioning of
human brains and components thereof.
Most neural networks have some sort of "training" rule whereby the weights
of connections are adjusted on the basis of presented patterns. In other words,
neural networks "learn" from examples, just like children learn to recognize
dogs from examples of dogs, and exhibit some structural capability for
generalization.
Neural networks normally have great potential for parallelism, since the
computations of the components are independent of each other.
What can you do with a Neural Network and what not?
In principle, NNs can compute any computable function, i.e. they can do
everything a normal digital computer can do. Especially anything that can be
represented as a mapping between vector spaces can be approximated to
arbitrary precision by feedforward NNs (which is the most often used type).
In practice, NNs are especially useful for mapping problems which are
tolerant of some errors, have lots of example data available, but to which hard
and fast rules can not easily be applied. NNs are, at least today, difficult to
apply successfully to problems that concern manipulation of symbols and
memory.
Who is concerned with Neural Networks?
Neural Networks are interesting for quite a lot of very dissimilar people:
o Computer scientists want to find out about the properties of
non-symbolic information processing with neural nets and about
learning systems in general.
o Engineers of many kinds want to exploit the capabilities of neural
networks on many areas (e.g. signal processing) to solve their
application problems.
o Cognitive scientists view neural networks as a possible apparatus to
describe models of thinking and conscience (High-level brain
function).
o Neuro-physiologists use neural networks to describe and explore
medium-level brain function (e.g. memory, sensory system, motorics).
o Physicists use neural networks to model phenomena in statistical
mechanics and for a lot of other tasks.
o Biologists use Neural Networks to interpret nucleotide sequences.
o Philosophers and some other people may also be interested in Neural
Networks for various reasons.
How many learning methods for NNs exist? Which?
There are many many learning methods for NNs by now. Nobody knows
exactly how many. New ones (at least variations of existing ones) are invented
every week. Below is a collection of some of the most well known methods;
not claiming to be complete.
The main categorization of these methods is the distinction of supervised from
unsupervised learning:
In supervised learning, there is a "teacher" who in the learning phase "tells"
the net how well it performs ("reinforcement learning") or what the correct
behavior would have been ("fully supervised learning").
In unsupervised learning the net is autonomous: it just looks at the data it is
presented with, finds out about some of the properties of the data set and
learns to reflect these properties in its output. What exactly these properties
are, that the network can learn to recognise, depends on the particular network
model and learning method.
Many of these learning methods are closely connected with a certain (class of)
network topology.
Now here is the list, just giving some names:
1. UNSUPERVISED LEARNING (i.e. without a "teacher"):
1). Feedback Nets:
a). Additive Grossberg (AG)
b). Shunting Grossberg (SG)
c). Binary Adaptive Resonance Theory (ART1)
d). Analog Adaptive Resonance Theory (ART2, ART2a)
e). Discrete Hopfield (DH)
f). Continuous Hopfield (CH)
g). Discrete Bi-directional Associative Memory (BAM)
h). Temporal Associative Memory (TAM)
i). Adaptive Bi-directional Associative Memory (ABAM)
j). Kohonen Self-organizing Map/Topology-preserving map (SOM/TPM)
k). Competitive learning
2). Feedforward-only Nets:
a). Learning Matrix (LM)
b). Driver-Reinforcement Learning (DR)
c). Linear Associative Memory (LAM)
d). Optimal Linear Associative Memory (OLAM)
e). Sparse Distributed Associative Memory (SDM)
f). Fuzzy Associative Memory (FAM)
g). Counterprogation (CPN)
2. SUPERVISED LEARNING (i.e. with a "teacher"):
1). Feedback Nets:
a). Brain-State-in-a-Box (BSB)
b). Fuzzy Cognitive Map (FCM)
c). Boltzmann Machine (BM)
d). Mean Field Annealing (MFT)
e). Recurrent Cascade Correlation (RCC)
f). Learning Vector Quantization (LVQ)
g). Backpropagation through time (BPTT)
h). Real-time recurrent learning (RTRL)
i). Recurrent Extended Kalman Filter (EKF)
2). Feedforward-only Nets:
a). Perceptron
b). Adaline, Madaline
c). Backpropagation (BP)
d). Cauchy Machine (CM)
e). Adaptive Heuristic Critic (AHC)
f). Time Delay Neural Network (TDNN)
g). Associative Reward Penalty (ARP)
h). Avalanche Matched Filter (AMF)
i). Backpercolation (Perc)
j). Artmap
k). Adaptive Logic Network (ALN)
l). Cascade Correlation (CasCor)
m). Extended Kalman Filter(EKF)
Good introductory literature about Neural Networks?
1.) The best (subjectively, of course -- please don't flame me):
Haykin, S. (1994). Neural Networks, a Comprehensive Foundation.
Macmillan, New York, NY. "A very readable, well written intermediate to
advanced text on NNs Perspective is primarily one of pattern recognition,
estimation and signal processing. However, there are well-written chapters on
neurodynamics and VLSI implementation. Though there is emphasis on
formal mathematical models of NNs as universal approximators, statistical
estimators, etc., there are also examples of NNs used in practical applications.
The problem sets at the end of each chapter nicely complement the material. In
the bibliography are over 1000 references. If one buys only one book on neural
networks, this should be it."
Hertz, J., Krogh, A., and Palmer, R. (1991). Introduction to the Theory of
Neural Computation. Addison-Wesley: Redwood City, California. ISBN
0-201-50395-6 (hardbound) and 0-201-51560-1 (paperbound) Comments:
"My first impression is that this one is by far the best book on the topic. And
it's below $30 for the paperback."; "Well written, theoretical (but not
overwhelming)"; It provides a good balance of model development,
computational algorithms, and applications. The mathematical derivations are
especially well done"; "Nice mathematical analysis on the mechanism of
different learning algorithms"; "It is NOT for mathematical beginner. If you
don't have a good grasp of higher level math, this book can be really tough to
get through."
Masters,Timothy (1994). Practical Neural Network Recipes in C++. Academic
Press, ISBN 0-12-479040-2, US $45 incl. disks. "Lots of very good practical
advice which most other books lack."
2.) Books for the beginner:
Aleksander, I. and Morton, H. (1990). An Introduction to Neural Computing.
Chapman and Hall. (ISBN 0-412-37780-2). Comments: "This book seems to
be intended for the first year of university education."
Beale, R. and Jackson, T. (1990). Neural Computing, an Introduction. Adam
Hilger, IOP Publishing Ltd. : Bristol. (ISBN 0-85274-262-2). Comments:
"It's clearly written. Lots of hints as to how to get the adaptive models covered
to work (not always well explained in the original sources). Consistent
mathematical terminology. Covers perceptrons, error-backpropagation,
Kohonen self-org model, Hopfield type models, ART, and associative
memories."
Dayhoff, J. E. (1990). Neural Network Architectures: An Introduction. Van
Nostrand Reinhold: New York. Comments: "Like Wasserman's book,
Dayhoff's book is also very easy to understand".
Fausett, L. V. (1994). Fundamentals of Neural Networks: Architectures,
Algorithms and Applications, Prentice Hall, ISBN 0-13-334186-0. Also
published as a Prentice Hall International Edition, ISBN 0-13-042250-9.
Sample software (source code listings in C and Fortran) is included in an
Instructor's Manual. "Intermediate in level between Wasserman and
Hertz/Krogh/Palmer. Algorithms for a broad range of neural networks,
including a chapter on Adaptive Resonance Theory with ART2. Simple
examples for each network."
Freeman, James (1994). Simulating Neural Networks with Mathematica,
Addison-Wesley, ISBN: 0-201-56629-X. Helps the reader make his own
NNs. The mathematica code for the programs in the book is also available
through the internet: Send mail to MathSource@wri.com or try
http://www.wri.com/ on the World Wide Web.
Hecht-Nielsen, R. (1990). Neurocomputing. Addison Wesley. Comments: "A
good book", "comprises a nice historical overview and a chapter about NN
hardware. Well structured prose. Makes important concepts clear."
McClelland, J. L. and Rumelhart, D. E. (1988). Explorations in Parallel
Distributed Processing: Computational Models of Cognition and Perception
(software manual). The MIT Press. Comments: "Written in a tutorial style,
and includes 2 diskettes of NN simulation programs that can be compiled on
MS-DOS or Unix (and they do too !)"; "The programs are pretty reasonable as
an introduction to some of the things that NNs can do."; "There are *two*
editions of this book. One comes with disks for the IBM PC, the other comes
with disks for the Macintosh".
McCord Nelson, M. and Illingworth, W.T. (1990). A Practical Guide to
Neural Nets. Addison-Wesley Publishing Company, Inc. (ISBN
0-201-52376-0). Comments: "No formulas at all"; "It does not have much
detailed model development (very few equations), but it does present many
areas of application. It includes a chapter on current areas of research. A
variety of commercial applications is discussed in chapter 1. It also includes a
program diskette with a fancy graphical interface (unlike the PDP diskette)".
Muller, B. and Reinhardt, J. (1990). Neural Networks, An Introduction.
Springer-Verlag: Berlin Heidelberg New York (ISBN: 3-540-52380-4 and
0-387-52380-4). Comments: The book was developed out of a course on
neural-network models with computer demonstrations that was taught by the
authors to Physics students. The book comes together with a PC-diskette. The
book is divided into three parts: (1) Models of Neural Networks; describing
several architectures and learning rules, including the mathematics. (2)
Statistical Physics of Neural Networks; "hard-core" physics section
developing formal theories of stochastic neural networks. (3) Computer Codes;
explanation about the demonstration programs. First part gives a nice
introduction into neural networks together with the formulas. Together with
the demonstration programs a 'feel' for neural networks can be developed.
Orchard, G.A. & Phillips, W.A. (1991). Neural Computation: A Beginner's
Guide. Lawrence Earlbaum Associates: London. Comments: "Short
user-friendly introduction to the area, with a non-technical flavour.
Apparently accompanies a software package, but I haven't seen that yet".
Rao, V.B & H.V. (1993). C++ Neural Networks and Fuzzy Logic. MIS:Press,
ISBN 1-55828-298-x, US $45 incl. disks. "Probably not 'leading edge' stuff
but detailed enough to get your hands dirty!"
Wasserman, P. D. (1989). Neural Computing: Theory & Practice. Van
Nostrand Reinhold: New York. (ISBN 0-442-20743-3) Comments:
"Wasserman flatly enumerates some common architectures from an engineer's
perspective ('how it works') without ever addressing the underlying
fundamentals ('why it works') - important basic concepts such as clustering,
principal components or gradient descent are not treated. It's also full of
errors, and unhelpful diagrams drawn with what appears to be PCB board
layout software from the '70s. For anyone who wants to do active research in
the field I consider it quite inadequate"; "Okay, but too shallow"; "Quite easy
to understand"; "The best bedtime reading for Neural Networks. I have given
this book to numerous colleagues who want to know NN basics, but who never
plan to implement anything. An excellent book to give your manager."
Wasserman, P.D. (1993). Advanced Methods in Neural Computing. Van
Nostrand Reinhold: New York (ISBN: 0-442-00461-3). Comments: Several
neural network topics are discussed e.g. Probabilistic Neural Networks,
Backpropagation and beyond, neural control, Radial Basis Function Networks,
Neural Engineering. Furthermore, several subjects related to neural networks
are mentioned e.g. genetic algorithms, fuzzy logic, chaos. Just the
functionality of these subjects is described; enough to get you started. Lots of
references are given to more elaborate descriptions. Easy to read, no extensive
mathematical background necessary.
3. Dedicated Neural Network Journals:
Title: Neural Networks
Publish: Pergamon Press
Address: Pergamon Journals Inc., Fairview Park, Elmsford,
New York 10523, USA and Pergamon Journals Ltd.
Headington Hill Hall, Oxford OX3, 0BW, England
Freq.: 10 issues/year (vol. 1 in 1988)
Cost/Yr: Free with INNS or JNNS or ENNS membership ($45?),
Individual $65, Institution $175
ISSN #: 0893-6080
Remark: Official Journal of International Neural Network Society (INNS),
European Neural Network Society (ENNS) and Japanese Neural
Network Society (JNNS).
Contains Original Contributions, Invited Review Articles, Letters
to Editor, Book Reviews, Editorials, Announcements, Software Surveys.
Title: Neural Computation
Publish: MIT Press
Address: MIT Press Journals, 55 Hayward Street Cambridge,
MA 02142-9949, USA, Phone: (617) 253-2889
Freq.: Quarterly (vol. 1 in 1989)
Cost/Yr: Individual $45, Institution $90, Students $35; Add $9 Outside USA
ISSN #: 0899-7667
Remark: Combination of Reviews (10,000 words), Views (4,000 words)
and Letters (2,000 words). I have found this journal to be of
outstanding quality.
(Note: Remarks supplied by Mike Plonski "plonski@aero.org")
Title: IEEE Transactions on Neural Networks
Publish: Institute of Electrical and Electronics Engineers (IEEE)
Address: IEEE Service Cemter, 445 Hoes Lane, P.O. Box 1331, Piscataway, NJ,
08855-1331 USA. Tel: (201) 981-0060
Cost/Yr: $10 for Members belonging to participating IEEE societies
Freq.: Quarterly (vol. 1 in March 1990)
Remark: Devoted to the science and technology of neural networks
which disclose significant technical knowledge, exploratory
developments and applications of neural networks from biology to
software to hardware. Emphasis is on artificial neural networks.
Specific aspects include self organizing systems, neurobiological
connections, network dynamics and architecture, speech recognition,
electronic and photonic implementation, robotics and controls.
Includes Letters concerning new research results.
(Note: Remarks are from journal announcement)
Title: International Journal of Neural Systems
Publish: World Scientific Publishing
Address: USA: World Scientific Publishing Co., 1060 Main Street, River Edge,
NJ 07666. Tel: (201) 487 9655; Europe: World Scientific Publishing
Co. Ltd., 57 Shelton Street, London WC2H 9HE, England.
Tel: (0171) 836 0888; Asia: World Scientific Publishing Co. Pte. Ltd.,
1022 Hougang Avenue 1 #05-3520, Singapore 1953, Rep. of Singapore
Tel: 382 5663.
Freq.: Quarterly (Vol. 1 in 1990)
Cost/Yr: Individual $122, Institution $255 (plus $15-$25 for postage)
ISSN #: 0129-0657 (IJNS)
Remark: The International Journal of Neural Systems is a quarterly
journal which covers information processing in natural
and artificial neural systems. Contributions include research papers,
reviews, and Letters to the Editor - communications under 3,000
words in length, which are published within six months of receipt.
Other contributions are typically published within nine months.
The journal presents a fresh undogmatic attitude towards this
multidisciplinary field and aims to be a forum for novel ideas and
improved understanding of collective and cooperative phenomena with
computational capabilities.
Papers should be submitted to World Scientific's UK office. Once a
paper is accepted for publication, authors are invited to e-mail
the LaTeX source file of their paper in order to expedite publication.
Title: International Journal of Neurocomputing
Publish: Elsevier Science Publishers, Journal Dept.; PO Box 211;
1000 AE Amsterdam, The Netherlands
Freq.: Quarterly (vol. 1 in 1989)
Editor: V.D. Sanchez A.; German Aerospace Research Establishment;
Institute for Robotics and System Dynamics, 82230 Wessling, Germany.
Current events and software news editor: Dr. F. Murtagh, ESA,
Karl-Schwarzschild Strasse 2, D-85748, Garching, Germany,
phone +49-89-32006298, fax +49-89-32006480, email fmurtagh@eso.org
Title: Neural Processing Letters
Publish: D facto publications
Address: 45 rue Masui; B-1210 Brussels, Belgium
Phone: (32) 2 245 43 63; Fax: (32) 2 245 46 94
Freq: 6 issues/year (vol. 1 in September 1994)
Cost/Yr: BEF 4400 (about $140)
ISSN #: 1370-4621
Remark: The aim of the journal is to rapidly publish new ideas, original
developments and work in progress. Neural Processing Letters
covers all aspects of the Artificial Neural Networks field.
Publication delay is about 3 months.
FTP server available:
ftp://ftp.dice.ucl.ac.be/pub/neural-nets/NPL.
WWW server available:
http://www.dice.ucl.ac.be/neural-nets/NPL/NPL.html
Title: Neural Network News
Publish: AIWeek Inc.
Address: Neural Network News, 2555 Cumberland Parkway, Suite 299,
Atlanta, GA 30339 USA. Tel: (404) 434-2187
Freq.: Monthly (beginning September 1989)
Cost/Yr: USA and Canada $249, Elsewhere $299
Remark: Commericial Newsletter
Title: Network: Computation in Neural Systems
Publish: IOP Publishing Ltd
Address: Europe: IOP Publishing Ltd, Techno House, Redcliffe Way, Bristol
BS1 6NX, UK; IN USA: American Institute of Physics, Subscriber
Services 500 Sunnyside Blvd., Woodbury, NY 11797-2999
Freq.: Quarterly (1st issue 1990)
Cost/Yr: USA: $180, Europe: 110 pounds
Remark: Description: "a forum for integrating theoretical and experimental
findings across relevant interdisciplinary boundaries." Contents:
Submitted articles reviewed by two technical referees paper's
interdisciplinary format and accessability." Also Viewpoints and
Reviews commissioned by the editors, abstracts (with reviews) of
articles published in other journals, and book reviews.
Comment: While the price discourages me (my comments are based
upon a free sample copy), I think that the journal succeeds
very well. The highest density of interesting articles I
have found in any journal.
(Note: Remarks supplied by kehoe@csufres.CSUFresno.EDU)
Title: Connection Science: Journal of Neural Computing,
Artificial Intelligence and Cognitive Research
Publish: Carfax Publishing
Address: Europe: Carfax Publishing Company, P. O. Box 25, Abingdon,
Oxfordshire OX14 3UE, UK. USA: Carafax Publishing Company,
85 Ash Street, Hopkinton, MA 01748
Freq.: Quarterly (vol. 1 in 1989)
Cost/Yr: Individual $82, Institution $184, Institution (U.K.) 74 pounds
Title: International Journal of Neural Networks
Publish: Learned Information
Freq.: Quarterly (vol. 1 in 1989)
Cost/Yr: 90 pounds
ISSN #: 0954-9889
Remark: The journal contains articles, a conference report (at least the
issue I have), news and a calendar.
(Note: remark provided by J.R.M. Smits "anjos@sci.kun.nl")
Title: Sixth Generation Systems (formerly Neurocomputers)
Publish: Gallifrey Publishing
Address: Gallifrey Publishing, PO Box 155, Vicksburg, Michigan, 49097, USA
Tel: (616) 649-3772, 649-3592 fax
Freq. Monthly (1st issue January, 1987)
ISSN #: 0893-1585
Editor: Derek F. Stubbs
Cost/Yr: $79 (USA, Canada), US$95 (elsewhere)
Remark: Runs eight to 16 pages monthly. In 1995 will go to floppy disc-based
publishing with databases +, "the equivalent to 50 pages per issue are
planned." Often focuses on specific topics: e.g., August, 1994 contains two
articles: "Economics, Times Series and the Market," and "Finite Particle
Analysis - [part] II." Stubbs also directs the company Advanced Forecasting
Technologies. (Remark by Ed Rosenfeld: ier@aol.com)
Title: JNNS Newsletter (Newsletter of the Japan Neural Network Society)
Publish: The Japan Neural Network Society
Freq.: Quarterly (vol. 1 in 1989)
Remark: (IN JAPANESE LANGUAGE) Official Newsletter of the Japan Neural
Network Society(JNNS)
(Note: remarks by Osamu Saito "saito@nttica.NTT.JP")
Title: Neural Networks Today
Remark: I found this title in a bulletin board of october last year.
It was a message of Tim Pattison, timpatt@augean.OZ
(Note: remark provided by J.R.M. Smits "anjos@sci.kun.nl")
Title: Computer Simulations in Brain Science
Title: Internation Journal of Neuroscience
Title: Neural Network Computation
Remark: Possibly the same as "Neural Computation"
Title: Neural Computing and Applications
Freq.: Quarterly
Publish: Springer Verlag
Cost/yr: 120 Pounds
Remark: Is the journal of the Neural Computing Applications Forum.
Publishes original research and other information
in the field of practical applications of neural computing.
GO TO
Science and Technology Papers / HOME |