The document describes a version of Don Knuth's WEB system, adapted to Perl by Thomas Tensi. Knuth's original Pascal programs have been reimplemented in Perl to take advantage of features offered by Perl.

Readers who are familiar with Knuth's memo "The WEB System of Structured Documentation" will be able to skim this material rapidly, because PerlWEB is essentially a very much simplified subset of WEB. PerlWEB does not need WEB's features for macro definition, because it uses a C preprocessor instead to take care of macros. Similarly, the WEB conventions of denoting octal and hexadecimal constants by @'77 and @"3f are omitted. Since PerlWEB does no prettyprinting of code parts a lot of WEB notations can also be thrown out. Many other features of WEB have nevertheless been retained.

Introduction

The philosophy behind WEB is that an experienced system programmer, who wants to provide the best possible documentation of software products, needs two things simultaneously: a language for formatting (like TeX or HTML), and a language like Perl for programming. Neither type of language can provide the best documentation by itself. But when both are appropriately combined, we obtain a system that is much more useful than either language separately.

The structure of a software program may be thought of as a "web" that is made up of many interconnected pieces. To document such a program, we want to explain each individual part of the web and how it relates to its neighbors. The typographic tools provided by the formatiing language give us an opportunity to explain the local structure of each part by making that structure visible, and the programming tools provided by \Cee\ make it possible for us to specify the algorithms formally and unambiguously. By combining the two, we can develop a style of programming that maximizes our ability to perceive the structure of a complex piece of software, and at the same time the documented programs can be mechanically translated into a working software system that matches the documentation.

The WEB system consists of two programs named w2h and w2p. When writing a PerlWEB program the user keeps the Perl code and the documentation in the same file, called the WEB file and generally named something.web. The command `w2h something.web -o something.htm' creates an output file something.htm, which can then be fed to some HTML browser, yielding a ``pretty printed'' version of something.web. The typeset output also includes extensive cross-index information and hyperlinking that is gathered automatically. Similarly, if you run the command `w2p something.web -o something.prl' you will get a Perl file something.prl, with can then run through Perl.

Besides providing a documentation tool, WEB enhances the Perl language by providing the ability to permute pieces of the program text, so that a large system can be understood entirely in terms of small modules and their local interrelationships. The w2p program takes a given web and moves the modules from their web structure into the order required by Perl; the advantage of programming in WEB is that the algorithms can be expressed in a form where each module is explained separately. The w2h program takes a given web and intertwines the documentation and Perl portions contained in each module.

A user of WEB should be fairly familiar with the Perl programming language. A minimal amount of acquaintance with HTML is also desirable, but in fact it can be acquired as one uses WEB, since straight text can be typeset with virtually no knowledge of that language. To someone familiar with both Perl and HTML the amount of effort necessary to learn the commands of WEB is small.

The language

Two kinds of material go into WEB files: HTML text and Perl text. A programmer writing in WEB should be thinking both of the documentation and of the Perl program being created; i.e., the programmer should be instinctively aware of the different actions that w2h and w2p will perform on the WEB file. HTML text is essentially copied without change by w2h, and it is entirely deleted by w2p; the HTML text is "pure documentation". Perl text is shuffled around by w2p, according to rules that will become clear later, and essentially kept untouched by w2h. For now the important point to keep in mind is that there are two kinds of text. Writing WEB programs is something like writing HTML documents, but with an additional "Perl mode" that is added to HTML.

A WEB file is built up from units called modules that are more or less self-contained. Each module has three parts:

1. A HTML part, containing explanatory material about what is going on in the module.
2. A definition part, containing macro definitions that serve as abbreviations for Perl constructions that would be less comprehensible if written out in full each time. They are turned by w2p into preprocessor macro definitions.
3. A Perl part, containing a piece of the program that w2p will produce. This Perl code should ideally be about a dozen lines long, so that it is easily comprehensible as a unit and so that its structure is readily perceived.

The three parts of each module must appear in this order; i.e., the HTML commentary must come first, then the definitions, and finally the Perl code. Any of the parts may be empty.

A module begins with either of the symbols `#_' or `#*', where `_' denotes a blank space. A module ends at the beginning of the next module (i.e., at the next `#_' or `#*'), or at the end of the file, whichever comes first. The WEB file may also contain material that is not part of any module at all, namely the text (if any) that occurs before the first module. Such text is said to be ``in limbo''; it is ignored by w2p and w2h, so its function is to provide comments.

Modules are numbered consecutively, starting with 1. These numbers appear at the beginning of each module of the HTML documentation output by w2h, and they appear as bracketed comments at the beginning and end of the code generated by that module in the Perl program output by w2p.

Fortunately, you never mention these numbers yourself when you are writing in WEB. You just say `#_' or `#*' at the beginning of each new module, and the numbers are supplied automatically by w2h and w2p. As far as you are concerned, a module has a name instead of a number; its name is specified by writing `#<' followed by HTML text followed by `#>'. When w2h outputs a module name, it replaces the `#<' and `#>' by angle brackets and inserts the module number in small type. Thus, when you read the output of w2h it is easy to locate any module that is referred to in another module. This number is also a hyperlink anchored at the first definition of this module.

For expository purposes, a module name should be a good description of the contents of that module; i.e., it should stand for the abstraction represented by the module. Then the module can be ``plugged into'' one or more other modules in such a way that unimportant details of its inner workings are suppressed. A module name therefore ought to be long enough to convey the necessary meaning. Unfortunately, however, it is laborious to type such long names over and over again, and it is also difficult to specify a long name twice in exactly the same way so that w2h and w2p will be able to match the names to the modules. To ameliorate this difficulty, w2h and w2p let you abbreviate a module name after its first appearance in the WEB file; you can type simply `#<µ...#>', where µ is any string that is a prefix of exactly one module name appearing in the file. For example, `#<Clear the arrays#>' can be abbreviated to `#<Clear...#>' if no other module name begins with the five letters `Clear'. Module names must otherwise match character for character, except that consecutive blank spaces and/or tab marks are treated as equivalent to single spaces, and such spaces are deleted at the beginning and end of the name. Thus, `#< Clear   the arrays #>' will also match the name in the previous example.

We have said that a module begins with `#_' or `#*', but we didn't say how it gets divided up into a HTML part, a definition part, and a Perl part. The definition part begins with the first appearance of `#d' in the module, and the Perl part begins with the first appearance of `#p' or `#<'. The latter option `#<' stands for the beginning of a module name, which is the name of the module itself. An equals sign (=) must follow the `#>' at the end of this module name; you are saying, in effect, that the module name stands for the Perl text that follows, so you say `<module name>=Perl text'. Alternatively, if the Perl part begins with `#p' instead of a module name, the current module is said to be unnamed. Note that module names cannot appear in the definition part of a module, because the first `#<' in a module signals the beginning of its Perl part. But any number of module names might appear in the Perl part, once it has started.

The general idea of w2p is to make a Perl program out of these modules in the following way: First all the macro definitions indicated by `#d' are turned into preprocessor macro definitions and copied at the beginning. Then the Perl parts of unnamed modules are copied down, in order; this constitutes the initial approximation T₀ to the text of the program. (There should be at least one unnamed module, otherwise there will be no program.) Then all module names that appear in the initial text T₀ are replaced by the Perl parts of the corresponding modules, and this substitution process continues until no module names remain. All comments are removed, because the Perl program is intended only for the eyes of the Perl compiler.

If the same name has been given to more than one module, the Perl text for that name is obtained by putting together all of the Perl parts in the corresponding modules. This feature is useful, for example, in a module named `Global variables in the outer block', since one can then declare global variables in whatever modules those variables are introduced. When several modules have the same name, w2h assigns the first module number as the number corresponding to that name and hyperlinks the other from the module name of the first section telling the reader that there are some other extension modules.

The general idea of w2h is to make a HTML file from the WEB file in the following way: Each module is output in turn, possibly interspersed with delimiters between modules. The text parts are formatted, the code parts are taken as is. Finally, w2h will generate a cross-reference index that is an alphabetized list of the module names, as well as a table of contents that shows the module numbers for each ``starred'' module.

What is a ``starred'' module, you ask? A module that begins with `#*' instead of `#_' is slightly special in that it denotes a new major group of modules. The `#*' should be followed by the title of this group, followed by a period. Such modules will always start with a thick horizontal line in the HTML output. The title will also appear in the table of contents, and in boldface type at the beginning of its module.

The HTML output produced by w2h for each module consists of the following: First comes the module number (e.g., `123.' at the beginning of module 123, except that a thick horizontal rule is inserted at the beginning of a starred module). Then comes the text part of the module, copied almost verbatim except as noted below. Then comes the definition part and the Perl part, formatted so that there will be a little extra space between them if both are nonempty. The definition and Perl parts are copied as is apart from the fact that they are set in a special font.

When you are typing the text part, you will probably want to make frequent reference to variables and other quantities in your Perl code, and you will want those variables to have the same typographic treatment when they appear in your text as when they appear in your program. Therefore the PerlWEB language allows you to get the effect of Perl editing within the text part, if you place | marks before and after the Perl material. For example, suppose you want to say something like this:

If $p is assigned a reference as in $p=\@var, then $p->[0] denotes the first element of @var.

The text part would look like this in your PerlWEB file:

If |$p| is assigned a reference as in |$p=\@var|, then |$p->[0]| denotes the first element of |@var|.

And w2h translates this into something you are glad you didn't have to type:

If <tt>$p</tt> is assigned a reference as in <tt>$p=\@var</tt>, then <tt>$p-&gt;[0]</tt> denotes the first element of <tt>@var</tt>.

Macros

The control code #d followed by identifier=Perl text or by identifier(par₁,...,par_n)=Perl text (where there may not be any white space between the identifier and the parentheses in the second case) is transformed by w2p into a preprocessor command, starting with #define, which is printed at the top of the Perl output file as explained earlier.

A `#d' macro definition can go on for several lines, and the newlines don't have to be protected by backslashes, since w2p itself inserts the backslashes. If for any reason you need a #define command at a specific spot in your Perl file, you can treat it as Perl code, instead of as a PerlWEB macro; but then you do have to protect newlines yourself.

Control codes

A PerlWEB control code is a two-character combination of which the first is `#'. We've already seen the meaning of several control codes; here is a complete list of all of them.

The letters L, T, P and/or M following each code indicate whether or not that code is allowable in limbo, in HTML text, in Perl text and/or in module names. A slash before such a letter means that the control code terminates the present part of the WEB file; for example, /L means that this control code ends the limbo material before the first module. Any other text in brackets will be explained in the paragraph following.

#_ [/L,/P,/T]

This denotes the beginning of a new (unstarred) module. A tab mark or form feed or end-of-line character is equivalent to a space when it follows an # sign (and in most other cases).

#* [/L,/P,/T]

This denotes the beginning of a new starred module, i.e., a module that begins a new major group. The title of the new group should appear after the #*, followed by a period. HTML control sequences should be avoided in such titles unless they are quite simple. The very first module should be starred.

#d [/P,/T]

Macro definitions begin with #d (or #D), followed by an identifier and optional parameters and Perl text as explained earlier.

#p [/P,/T]

The Perl part of an unnamed module begins with #p (or #P). This causes w2p to append the following Perl code to the initial program text T₀ as explained above.

Because of the rules by which every module is broken into three parts, the control codes `#d' and `#p' are not allowed to occur once the Perl part of a module has begun.

#< [P,/T]

A module name begins with #< followed by text followed by #>. The module name may be abbreviated, after its first appearance in a WEB file, by giving any unique prefix followed by ..., where the three dots immediately precede the closing #>. Module names may not appear in Perl text that is enclosed in |, nor may they appear in the definition part of a module (since the appearance of a module name ends the definition part and begins the Perl part).

#x #y #z [change_file]

w2h and w2p are designed to work with two input files, called web_file and change_file, where change_file contains data that overrides selected portions of web_file. The resulting merged text is actually what has been called the WEB file elsewhere in this report.

Here's how it works: The change file consists of zero or more ``changes,'' where a change has the form

    #x
      old lines
      ...
    #y
      new lines
      ...
    #z
    

The special control codes #x, #y, #z, which are allowed only in change files, must appear at the beginning of a line; the remainder of such a line is ignored. The <old lines> represent material that exactly matches consecutive lines of the web_file; the <new lines> represent zero or more lines that are supposed to replace the old. Whenever the first ``old line'' of a change is found to match a line in the web_file, all the other lines in that change must match too.

Between changes, before the first change, and after the last change, the change file can have any number of lines that do not begin with `#x', `#y', or `#z'. Such lines are bypassed and not used for matching purposes.

This dual-input feature is useful when working with a master WEB file that has been received from elsewhere (e.g., web2html.web or web2perl.web), when changes are desirable to customize the program for your local computer system. You will be able to debug your system-dependent changes without clobbering the master web file; and once your changes are working, you will be able to incorporate them readily into new releases of the master web file that you might receive from time to time.

#i [web_file]

Furthermore the web_file itself can be a combination of several files. When either w2h or w2p is reading a file and encounters the control code #i at the beginning of a line, it interrupts normal reading and start looking at the file named after the #i, much as the C preprocessor does when it encounters an #include line. After the included file has been entirely read, the program goes back to the next line of the original file. The file name following #i must be surrounded by " characters. Include files can nest. Change files cannot use the #i feature.

Running the programs

The UNIX and DOS command line for w2p is

w2p web_file.web [ change_file.ch ] [ -o outfile ]

and the same conventions apply to w2h:

w2h web_file.web [ change_file.ch ] [ -o outfile ]

If no outfile is given output goes either to STDOUT (in UNIX) or to junk.prl for w2p or junk.htm for w2h (in DOS). If no change file is specified, the change file is null (i.e. no changes are incorporated...).

Installation

The PerlWEB-distribution contains:

To install the distribution on your system you have to do the following

1. Expand the ZIP-file in some directory (DOS) or unzip and untar the Unix archive.
2. Put all Perl programs in some directory where they can be located by Perl.
3. The script files must be put in some directory on the path of the shell (in UNIX) or COMMAND.COM (in DOS).
4. Make sure that Perl (version 5) is accessible. The DOS version additionally needs a C preprocessor PERLCPP which must be in the DOS path.

After that installation PerlWEB is functional.

The original files web2perl.web and web2perl.dos together with DOS change files and necessary include files (general.web, web2xxx.web) are also included in the distribution. Whenever code has to be changed, don't do it in the Perl files...

Acknowledgments by the original authors

The authors Knuth and Levy wish to thank all who contributed suggestions and criticism to the development of CWEB. They are especially grateful to Norman Ramsey, from whom the code for multiple output files is borrowed, and who has made literate programming accessible to users of yet other languages by means of his SPIDER system [see Communications of the ACM, 32 (1989), 1051-1055].

Acknowledgments

The author thanks Johannes Berger for correction and valuable comments on PerlWEB and the PerlWEB programs w2p and w2h.

Appendices

Appendix A -- File format

The following is an excerpt of the file web2xxx.web, which contains routines shared by w2p and w2h. Note that some of the lines are indented to show the program structure. The indentation is ignored by w2p, but users may find that PerlWEB files are quite readable if they have some such indentation. Besides this indentation is directly used by w2h.

The reader should first compare Appendix A to Appendix B; then the same material should be compared to Appendices C and D.

#*Process change file.
Pass 2 traverses all the lines read and checks a change file for any
substitutions necessary to adapt a master WEB file to a specific
environment.

  #<Process change file#>=
    {
      local *changefile_handle;

      #<Open change file into |changefile_handle|; abort on failure#>
      #<Traverse change file and update |@input_line_list|#>

      close(changefile_handle);
    }

    
# The change file is opened via the |$change_file_name| given. The file
handle is returned in |changefile_handle|. If anything unexpected happens,
the program is stopped.

  #<Open change file...#>=
    {
      my BooleanType $success=open(changefile_handle, $change_file_name);
      if (!$success) {
	fatal_error("web2perl: cannot open change file $change_file_name: $!");
      } #endif
    }
      

# The change file is read line by line. In principle the change file
consists of substitutions embedded in #x#y#z lines. The #x and #y lines
brackets a sequence of lines which have to be exactly matched in the input
file. The lines bracketed by #y and #z are the replacement for the lines
matched. All other lines are not significant and can be used for comments.

Another important point is that the sequence of substitutions must be
applied in order; thus the input file can be traversed sequentially.


# The change file is read sequentially and parsed by a finite automaton.
Whenever a #x, #y, or #z line is found a transition to the next state
occurs. There are four states, which form - more or less - a cyclic state
sequence. When parsing a substitution on the change file, the program first
tries to find matching lines in |@input_line_list|. The indices are stored
in |$first_line_number| and |$last_line_number|. The replacement text is
accumulated in |@changed_input_line_list|. After the #z has been parsed, the
substitution is carried out on |@input_line_list|.

If the change file ends prematurely, an appropriate error message is given.

  #<Traverse change file...#>=
    {
      my StringListType @changed_input_line_list;
      my CardinalType $first_line_number=FIRST_INDEX;
      my StringType $line;
      my CardinalType $scan_mode=0;

      while ($line=<changefile_handle>) {
        ## kick out NEWLINE
        $line=~s/\r?\n$//;
	if ($line =~ "^[$WEB_escape_char]([xyz])") {
	  #<Process change file entry#>
	  if ($scan_mode<4) {
	    next;
	  } #endif
	}
	#<Handle changefile line according to |$scan_mode|#>
      } #endwhile
      if ($scan_mode>0) {
        warning("unexpected eof in change file");
      } #endif
    }

Appendix B -- Translation by w2p

Here's the portion of the Perl code generated by w2p that corresponds to Appendix A. Notice that modules 26 and 28 have been tangled into module 25.

Note that parts not described in Appendix A have been replaced by ''[...]''. The tools will give error messages whenever referenced modules are not defined in a WEB file.

    {
      local *changefile_handle;

      {
        my  $success=open(changefile_handle, $change_file_name);
        if (!$success) {
          fatal_error("web2perl: cannot open change file $change_file_name: $!");
        } #endif
      }

      {
        @changed_input_line_list;
        my  $first_line_number=$[;
        my  $line;
        my  $scan_mode=0;

        while ($line=<changefile_handle>) {
          $line=~s/\r?\n$//;
          if ($line =~ "^[$WEB_escape_char]([xyz])") {
	    [...]

            if ($scan_mode<4) {
              next;
            } #endif
          }
	  [...]
        } #endwhile
        if ($scan_mode>0) {
          warning("unexpected eof in change file");
        } #endif
      }

      close(changefile_handle);
    }

Appendix C -- Translation by w2h

w2h translates a WEB file into a HTML text with the following properties:

• Each section in the PerlWEB program gets a unique number.
• The documentation parts of a PerlWEB section are taken as HTML paragraphs. Whenever a blank line separates two lines, a new paragraph is started. Lines starting with a minus are taken as lines from an unordered list.
• The definition and code part of a section is set in a special font as is. The module names in module applications are completely expanded. Additionally they have the section number appended where the corresponding module definition can be found. For navigation there is also a hyperlink to this section.
• Subroutines are handled similarly.
• Finally a lexically sorted list of all module names with definitions and applications is appended.

<P><A NAME="section_25"><B>25. Process change file.</B></A>
Pass 2 traverses all the lines read and checks a change file for any
substitutions necessary to adapt a master WEB file to a specific
environment.
</P>

<PRE>
  &lt;&lt;Process change file&gt;&gt; =
    {
      local *changefile_handle;
      &lt;&lt;Open change file into |changefile_handle|; abort on failure&gt;&gt;<a href="#section_26"><SUB>26</SUB></A>
      &lt;&lt;Traverse change file and update |@input_line_list|&gt;&gt;<A HREF="#section_28"><SUB>28</SUB></A>
      close(changefile_handle);
    }
    
</PRE>
<P><A NAME="section_26"><HR NOSHADE>26. </A>The change file is opened via the <CODE>$change_file_name</CODE> given. The file
handle is returned in <CODE>changefile_handle</CODE>. If anything unexpected happens,
the program is stopped.
</P>

<PRE>
  &lt;&lt;Open change file into |changefile_handle|; abort on failure&gt;&gt; =
    {
      my BooleanType $success=open(changefile_handle, $change_file_name);
      if (!$success) {
	fatal_error<SUB>14</SUB>("web2perl: cannot open change file $change_file_name: $!");
      } #endif
    }
      
</PRE>
<P><A NAME="section_27"><HR NOSHADE>27. </A>The change file is read line by line. In principle the change file
consists of substitutions embedded in #x#y#z lines. The #x and #y lines
brackets a sequence of lines which have to be exactly matched in the input
file. The lines bracketed by #y and #z are the replacement for the lines
matched. All other lines are not significant and can be used for comments.
</P>

<P>Another important point is that the sequence of substitutions must be
applied in order; thus the input file can be traversed sequentially.
</P>

<P><A NAME="section_28"><HR NOSHADE>28. </A>The change file is read sequentially and parsed by a finite automaton.
Whenever a #x, #y, or #z line is found a transition to the next state
occurs. There are four states, which form - more or less - a cyclic state
sequence. When parsing a substitution on the change file, the program first
tries to find matching lines in <CODE>@input_line_list</CODE>. The indices are stored
in <CODE>$first_line_number</CODE> and <CODE>$last_line_number</CODE>. The replacement text is
accumulated in <CODE>@changed_input_line_list</CODE>. After the #z has been parsed, the
substitution is carried out on <CODE>@input_line_list</CODE>.
</P>

<P>If the change file ends prematurely, an appropriate error message is given.
</P>

<PRE>
  &lt;&lt;Traverse change file and update |@input_line_list|&gt;&gt; =
    {
      my StringListType @changed_input_line_list;
      my CardinalType $first_line_number=FIRST_INDEX;
      my StringType $line;
      my CardinalType $scan_mode=0;
      while ($line=<changefile_handle>) {
        ## kick out NEWLINE
        $line=~s/\r?\n$//;
	if ($line =~ "^[$WEB_escape_char]([xyz])") {
	  &lt;&lt;Process change file entry&gt;&gt;<SUB>30</SUB>
	  if ($scan_mode<4) {
	    next;
	  } #endif
	}
	&lt;&lt;Handle changefile line according to |$scan_mode|&gt;&gt;<SUB>31</SUB>
      } #endwhile
      if ($scan_mode>0) {
        warning<SUB>14</SUB>("unexpected eof in change file");
      } #endif
    }
</PRE>

Appendix D -- The final document

The final document for looks like this (complete documents for web2perl.web and web2html.web are also available):

25. Process change file. Pass 2 traverses all the lines read and checks a change file for any substitutions necessary to adapt a master WEB file to a specific environment.

  <<Process change file>> =
    {
      local *changefile_handle;
      <<Open change file into |changefile_handle|; abort on failure>>26
      <<Traverse change file and update |@input_line_list|>>28
      close(changefile_handle);
    }
    

  <<Open change file into |changefile_handle|; abort on failure>> =
    {
      my BooleanType $success=open(changefile_handle, $change_file_name);
      if (!$success) {
	fatal_error14("web2perl: cannot open change file $change_file_name: $!");
      } #endif
    }
      

Another important point is that the sequence of substitutions must be applied in order; thus the input file can be traversed sequentially.

If the change file ends prematurely, an appropriate error message is given.

  <<Traverse change file and update |@input_line_list|>> =
    {
      my StringListType @changed_input_line_list;
      my CardinalType $first_line_number=FIRST_INDEX;
      my StringType $line;
      my CardinalType $scan_mode=0;
      while ($line=<changefile_handle>) {
        ## kick out NEWLINE
        $line=~s/\r?\n$//;
	if ($line =~ "^[$WEB_escape_char]([xyz])") {
	  <<Process change file entry>>30
	  if ($scan_mode<4) {
	    next;
	  } #endif
	}
	<<Handle changefile line according to |$scan_mode|>>31
      } #endwhile
      if ($scan_mode>0) {
        warning14("unexpected eof in change file");
      } #endif
    }