Academic publishing on the World Wide Web: an investigation into the opinions of selected internet users and academics.

Academic publishing on the World Wide Web: an investigation into the opinions of selected internet users and academics.

The academic publication enterprise is currently at a watershed. In part, this is due to the influence and recent proliferation of electronic media, epitomised by the Internet. Anyone with ready access to the Internet can now publish material, with delivery evolving from a one-to-many to a many-to-many system. In academia there are concerns that this evolution will lower the quality of publications, with a subsequent bias against the electronic medium of delivery. If this proves to have substance, there are important ramifications for the academic community, as publications are considered a currency with respect to citation value for promotions, funding of grants, etc. To investigate this potential bias two surveys were conducted. The first survey (which was qualitative) requested the opinions of one group consisting of experienced Internet users with an interest in the World Wide Web, and another group consisting of academic departmental heads from around Australia. From 68 valid responses, consisting of approximately equal numbers from both groups, it was found that a majority did not publish or read academic journals on the Web. Users of Web journals tended to read outside their discipline more often than not. High quality articles were seen to be of lower quality and in lower quantity on the Web than in print based journals. Nonetheless, nearly all respondents believed that Web journals will be more widely read and cited in the future. The second survey (which was quantitative) asked a different group of experts to rate a Web paper in comparison with a print based one. The test was counterbalanced, in that half of the experts had the sources reversed. Although the number of respondents was small, the results implied that a bias against Web based publications relative to those in print may exist.

Introduction - The publication watershed

Publishing is undergoing a revolution. With the proliferation of access to the Internet (Net) and the advent of the World Wide Web (Web) authors have commenced publishing their own writing and are supplying it directly to the customer. One-to-many written communication has evolved into many-to-many communication.

Concerns have been raised (e.g. Schauder, 1994) that as intermediate steps in the publishing process are removed, so too are steps in quality control. This is of particular concern in the arena of academic publishing, where research has traditionally undergone stringent quality control measures through the process of peer review. To maintain the quality found in traditional, peer reviewed print based academic journals, some journals published on the Net, known as ejournals, have been developed with the expectation that submitted papers are subjected to a similar peer review system (e.g. IPCT-J - Interpersonal Computing and Technology EJournal). The process differs only in that it is generally more rapid due to the use of the Net to send papers to editors and reviewers via the Web or email.

Schauder (1994) surveyed 743 senior academics to investigate the contribution ejournals can make to research communication. Of the respondents, 39 per cent used the Net, with proportionately more from the sciences than from the arts, and 92 per cent of these used the technology for email. The next most common use of the Net was for finding references to articles (14 per cent) but less than 7 per cent had published on the Net through discussion fora known as newsgroups or another form of publishing electronically (epublishing). In reply to whether epublishing would have some recognition by the university award/promotion system, 33 per cent did not know, 54 per cent said yes or to some extent and only 12 per cent said no. In summary, Schauder (1994) found that academics believed that the epublishing of academic articles will lead to greater diversity and choice in the academic community, which would be beneficial to the development and distribution of knowledge.

Harnad (1995) noted that academics were sceptical of the advantages of epublishing as a result of the initial chaotic conditions of the Net. However, he asks where paper printing would be if people had judged the system on the products of Guttenberg and a team of linotype operators rather than those of Shakespeare and Newton. He stressed that research must be peer reviewed, and that electronic review saved travel, talking time and costs, and achieved speed, geographical scope and interaction at a scale greater than ever before.

Odlyzko (1994) predicted that traditional journals will disappear within 10-20 years, or at least be converted to an electronic version. This would be both essential and possible due to the growth in the size of the literature (the archive of academic publication doubles every decade) and the growth in technology (the speed of microprocessors doubles every 1.5 years).

There is a possibility that a bias against academic work published on the Net has arisen. This could be due to a perception of a large amount of poor quality written work being sent around the Net, perhaps a result of the ease of electronic publishing that has arisen this decade. This could also be said of print based publications, although there exist filtering mechanisms such as libraries and personal reputations. In addition, aspects of the system are new (for example the Web) and any society will always experience some degree of resistance to change. A bias against electronic journal publications (ejournals) would be surprising in light of the availability of high quality colour graphics, sounds, movies and even executable programs that can accompany the text on the Web, with the potential capacity to enhance communication. There have also been concerns raised about the availability of socially restricted or dangerous information such as pornography and racism, whereas the attractive aspects of the Web appear to have not made as bold a headline.

In the new field of academic epublishing, it is important to test whether a bias has arisen. A bias against work published on the Net would have major ramifications with respect to citation value of journals and subsequent promotions, employment opportunities and granting of research funds. In spite of plenty of work having been performed on setting up ejournals, assessing the current state of epublishing and even some assessment of opinions of academic electronic publications, there has been no quantitative assessment of high quality visual publications on the Web. An assessment must be obtained so that those in the epublishing field can better address potential biases and problems.

This study aimed to investigate qualitatively the opinions of academic publication from members of the academic and Net communities. It quantitatively assessed the opinions of experts of research articles published in ejournals on the Net and examined how these compared with their opinions of research articles published in print based journals. The focus of the project was on scholarly, peer reviewed scientific publications that appear entirely and uniquely on the Web.

Method - Designing a Web based project

Two surveys were conducted. The first qualitative one was designed to gauge the opinions of academic Heads and Net users of academic publications on the Web. This was intended to expand on previous investigations and provide a background against which to frame the second survey. The second, quantitative one was designed to objectively measure any bias associated with Web publications by asking selected experts to assess two articles from Web and print based sources. Details of the two surveys are given below.

Academic opinions part I: Qualitative survey

The aim of the initial survey was to evaluate respondents' opinions of Web based publications, and explore possible reasons for any bias. It was also intended to determine if indeed there is support for the hypothesis that there is a bias against Web based publications. The survey was designed to determine how many of the respondents read or publish material on the Web, how they rate Web based publications against those in print, and to help determine some of the reasons for any differences. Information required to form a profile of the respondent was also considered important, particularly in respect to how much experience they had with the Net. The survey was tested on a pilot group of ten interested people from both academia and the epublishing field. Many of the changes suggested by the pilot group were subsequently acted upon.

Target survey groups were selected on the Web. A notification of the Universal Resource Locater (URL) of the Web based survey (Torok, 1995) was initially sent to the Web discussion group, World Wide Web Development (WWWDEV). This newsgroup commenced in early 1995 and by the time the survey was advertised had over 600 members interested in the development of the Web as a educational tool (Hall, 1995). As a result of the disappointing response after the first week (12 replies), the address of the survey was also sent to two other newsgroups. It was subsequently seen that these involved more relevant groups for this project, as evidenced by personal messages sent through email once the survey had been announced. One of the newsgroups, Hyperjournal, contained a total membership of 712, having been commenced in April 1995. Most people (75 per cent) were academic-related (teaching staff, librarians, computer support personnel etc), with about 25 per cent having a commercial publishing background (Keown, 1995). No detailed information was available about the other group that was sent the survey address, recipients of the Inter-Personal Computing Technology Journal (IPCT-J), but it contained a wide range of members interested in electronic publication.

Copies of the survey were also sent to a total of 68 academic Heads at eight large and nine small Australian universities. Rather than sending only the survey Internet address, a paper copy of the survey was sent to academic Heads. This was intended to avoid a bias in the responses from Web and Internet users. There was a possibility that the survey was passed on to a subordinate (who supposedly would have more time to complete it) or to somebody with a more comprehensive knowledge of the Web, as one participant noted via email. However, the assumption was made that the survey was completed by a senior academic.

Academic opinions part II: Quantitative survey

The aim of this second part of the research project was to quantitatively assess the opinions of academic publications on the Web relative to those in print based journals. Two similar academic articles in a scientific field were selected, one that had been published on the Web and the other from a print based journal. Clues towards the source of the article were deleted, such as the journal, author and references. Other clues such as the font and layout were reformatted to be as similar as possible.

Papers from psychiatric journals were selected as this field of science has a number of well developed Web based journals (Williams and Appelquist, 1995). In addition, the names of Australian psychiatry experts - potential survey participants - were readily available from a register (Knox, 1993). The selection of the electronic article (Bennett and Aquila, 1995) was based on its brevity, lack of diagrams and few tables. Print based journals were then browsed using CD-ROM technology to find a paper on a comparable subject, of similar length, content, with a lack of diagrams and tables. One print based journal article fitting these criteria was found (Caton et al., 1990).

All Australian doctors listed as specialising in psychiatry were selected from Knox (1993). Names and contact numbers were only available for 26 General Practitioners (GPs) and 14 University Professors. Thirteen GPs and seven University Professors (defined as Group A) were sent papers correctly identified as to their source (viz. Print or web) and Thirteen GPs and seven University Professors (were selected Group B) were sent pairs of incorrectly identified papers. The selection of which recipients received the correctly or incorrectly identified papers was random.

Respondents were asked to rate the quality of each article individually, rather than to make a comparison of the two articles directly. This is because respondents generally hesitate to rate two similar articles differently, due to their belief that the papers are supposed to be similar, or that they will be seen to be ignorant. Respondents were asked to give their judgement of the quality of the papers, rather than their opinion of them. As Trevitt et al. (1995) outline, an informed judgemental assessment is a holistic task, taking into account the possibility of alternative views in the outside world. Linacre (1992) recommends a numerical assessment based on accuracy (the truth of the content and numeracy of presentation), breadth (the adequacy of covering the topic), clarity (fluency within a lucid structure), depth (the extent to which matters are explained rather than simply mentioned), and enterprise (the element of creative thought). An overall judgement was also requested.

Results and discussion

Survey 1: Qualitative

In total, 67 people from both the academic Heads and Web user groups replied. Fifty-two per cent of the responses (35) were from academic Heads through the postal mail service, and 48 per cent of responses (32) were via automated reply through the Web. The response rate was much better from the group of academic Heads through the snail mail, with 51 per cent of those contacted replying (35 out of 68). Fewer than 5 per cent (32 out of at least 1 500) of Web users replied via the automatic response survey posted on the Web (Torok, 1995). One possible reason for the difference in reply success is that in receiving a survey in the mail, a person may feel more obliged to match the effort expended in delivering the survey by mailing back their reply. On the other hand, Web users may not bother to reply to a survey that is displayed passively at a particular Web site. The Web users who did reply were highly self-selected, in that they would need some motivation or reason to reply to the survey.

In answer to question 1 (Do you publish on the Web?) 37 per cent of respondents replied yes and 63 per cent replied no. In other words, nearly two thirds of the people surveyed had never published an article, home page, etc. on the Web. There may have been some confusion as to the definition of publish, as the survey did not specify what form of document was to have been published. Possible confusion may also have existed in that publication on any form of electronic media, such as the Net, may have been seen to be a Web publication. Of the academic Heads, approximately 1 in 10 had published on the Web, and over 9 out of 10 had never published. This could be due to a lack of suitable forum, or lack of time to surf the Net and obtain a feeling of comfort with the technology, or even due to conscious objection to the system. It could also be a function of age category and publishing experience. Further research is required to draw any conclusions from the reasons for such a small percentage of Web publishers amongst academic Heads. A much larger percentage of Web users had published on the Web, with 69 per cent having had a publication on the Web, and only 31 per cent not currently publishing.

Question 2 (Do you read academic journals on the Web?) was more specific about the subject matter on the Web that was of interest. Nearly half of all respondents had read academic journals on the Web, with 54 per cent not reading (and probably never having read) academic journals on the Web. Of the academic Heads, 20 per cent use the Web as a resource for academic journals, with the great majority preferring to stick to print based journals in their research. Of the Web users who replied, three quarters read academic journals on the Web with only 25 per cent not reading them. It is not surprising that a large percentage of Web users surveyed read academic journals on the Web, as the groups alerted to the location of the survey were actively involved in the epublishing field through newsgroups about Web publication.

The Web is used as an interdisciplinary tool more often than not, with 27 respondents replying positively to question 3 (Do you read articles on the Web outside your own area of research?). Fourteen of the respondents replied no, and 26 gave no response. It was more common for an academic Head to not read a wide range of articles on the Web than for them to read outside their area of research. However, this probably reflects a lack of Web use rather than a lack of interdisciplinary research aims.

As questions three to eight were optional for the group who had neither read nor published on the Web, about half of the respondents left these answer boxes blank. Although more than half of the respondents did answer these questions (perhaps due to interest or perhaps due to a misunderstanding of the instructions), the analysis of the results for these questions will focus on those who had either read or published on the Web. This is because the questions address the difference between the two publication media and those with experience of the Web would be likely to furnish a more reliable comparison of Web journals to print based journals. The group with less experience will be ignored because a comparison between something well known and something unknown would not be as valid. A total of 34 people answered yes to either questions one or two, and hence for questions three to eight their responses were considered. This included seven from the group of academic Heads and 27 from the group of Web users.

Question 4 asked for the respondent's general opinion of material published on the Web compared to that published in print based journals. High quality, peer reviewed research was generally seen to be of equal quality in most (20) cases. However, eight people saw high quality Web based articles as inferior to high quality print based articles, whereas only four respondents considered Web based articles to be better. None of the respondents considered the difference to be so great for Web material to be much better or much worse than print based material. Two respondents gave no response. For lower quality, but still peer reviewed research, most people (13) still considered the quality to be the same across both media. However, nine people considered low quality articles on the Web to be better (and in one case much better) than low quality research in print based journals, and eight respondents considered print based articles to be better. Three respondents gave no response for this question. Although the numbers were similar, fewer people considered print based low quality research to be better than Web based low quality research. The answers to the two separate parts of this question indicate that more people see high quality articles as inferior on the Web. Another interpretation could be that respondents saw Web based articles as the same or better than print based articles, however this conclusion is drawn from a consideration of both parts of the question together, which was not intended in the design of the question.

Answers to question 5, however, would seem to indicate that the latter of these two interpretations is invalid. This is because a bias against the Web, similar to the qualitative bias in question 4, appears in the opinions of the quantity of high quality material on the Web. Question 5 asked respondents to rate the relative abundance of high and low quality research in the two media. Many (24) people considered there to be a low abundance of high quality research in Web journals, whereas nobody considered there to be a lack of high quality research in print based journals. Similarly, many (19) people consider there to be a large amount of high quality research in print based journals, but very few (2) consider there to be much high quality research in Web based journals. 19 people replied that there was a medium amount of high quality research on both media. It is interesting to bear in mind that these respondents have had experience in either reading or publishing on the Web. With respect to low quality research, many (13) people considered there to be a low abundance of such material in print based journals, and twelve respondents considered there to be a high abundance of low quality material on the Web. Two respondents gave no response. A large number of people (31) replied that there was a medium amount of low quality research in both media.

Question 6a asked respondents, who considered papers from the two media to differ in quality, to suggest reasons for the difference. Four academic Heads replied that there were as yet no ejournals in their field and therefore they had no choice but to read print based journals only. Fifteen of the respondents who considered Web based journals to be of lower quality than those in print suggested a number of reasons for the difference. The most common reason was a passive one, in that due to their audience having a bias against, or not reading, ejournals they did not consider it worthwhile to publish their best work on the Web. Other respondents expressed concern that peer review was the exception rather than the rule at this stage of ejournal development. Other reasons given for Web based journals being of lower quality than those in print included their content, lack of permanence, lack of uniqueness, lack of tenure reward and their poor appearance. Seven of those who considered Web based papers to be better than those in print also gave reasons. It was said that some subjects are better handled electronically, due to the graphics capability of the Web. Other reasons included timeliness and the visionary nature of the Web. However, this question asked only for suggested reasons for the difference.

Perhaps more pertinent to this project is question 6b, which asked respondents which of the two media they were more likely to cite in their work. Fourteen respondents, mainly from the group of Web users, said they would cite either or both, depending on which was more relevant with respect to content. In other words, for this group of respondents the medium of delivery was irrelevant. Other reasons for citing both was to provide an access option for their readers to make finding the reference easier. It was commented that the situation may change in the future towards favouring more Web citations. Twenty-one respondents, the majority of them from the group of academic Heads, said they would cite only the print based medium. Reasons given were audience prejudice, no assurance of permanence, lack of access in developing countries, poor visual appearance and feel, and the difficulties of finding a correct referencing format. Only seven respondents, all from the group of Web users, said they would cite only the Web based journal article. The reasons given were the ease of access and the lack of physical space required for storage of journals.

Question 7 asked respondents to give their occupation and position. It was found that the majority of Web based respondents (60%) had a university teaching position, providing a measure of the validity of comparing the two survey groups. Although not all university teachers stated their position as Head, a majority were full professors. Therefore the two groups are comparable to some extent. However, it must be stressed that the group of Web users were highly self-selected. Other replies were returned from postgraduate students (11%), publishers and librarians (11%), consultants (11%) and others (7%).

From a speculative standpoint, replies to questions 8 and 9 imply that the situation could improve in the future with respect to acceptability of Web based journals. From question 8 (How long have you used the Web) it appeared that more people are exploring the Web as time progresses. There is a positive trend in the number of people using the Web, with more (29) people having joined the Web in the past year than have used the Web for one year (17), two years (15) or three or more years (6). The number of people not using the Web includes the 29 respondents who gave no response to this question. The answers to question 9 were very positive, with 95 per cent of respondents seeing publications on the Web becoming more widely read and cited in the future. Four respondents gave no response for this question.

Many respondents used the space provided for additional comments. Ten replies complimented the project, one in particular noting that projects such as this influence people's behaviour for the better and raise an awareness of Web journals. Six respondents made suggestions toward making the survey page more aesthetically pleasing. Comments relevant to the topic of epublishing were made, such as the need to overcome academic inertia and other problems such as copyright, permanence and visual layout. Problems with computer hardware and software were noted, as was the slow speed of loading Web documents due to communications bandwidth being too small for the large volume demands. Other comments were made that there was no time to explore the Web, or that it was easier to have documents sent through email or snail mail. One comment was made that such a survey was being conducted too soon, and that in another year the situation will be more appropriate. Accordingly, it would be interesting to repeat this project in another year or two to investigate whether the predictions of more widely used Web based academic publications have been realised.

In summary, the qualitative survey has provided a picture illustrating academic Head and Net user group's opinions of academic publications on the Web being of lower quality than those in print based publications. Hence it has provided evidence to justify the quantitative survey to measure this bias.

Survey II: Quantitative

Potential respondents were given three weeks to read the two papers and respond to the survey. Although the survey was designed to take less than half an hour to complete, only two replies were received by the designated deadline. The response rate was improved by personally telephoning each of the remaining participants. In total only six replies were received. Replies consisted of an assessment, as a mark out of ten, of each of the two papers. Four replies were received from Group A, which had the correct sources, (i.e. the Web and print based papers) and two were received from Group B, which had the sources swapped (i.e. the 'Web' - print based paper disguised as one from the Web - and 'print' - the Web based paper disguised as one from print - based papers). Five respondents were from the group of GPs and one respondent was from the group of academics. The latter two groups were too small to divide into separate academic and GP groups, therefore they were kept together in an overall group of experts in the field of psychiatry.

The amount or depth of each of the papers read was investigated. The majority of each paper was read in every case, so this will not be considered a limiting factor for the remainder of the analysis.

Quantitative assessment of article content

The assessments of the Web (or 'print') based paper relative to the other in the pair (i.e. the print or 'Web' based paper) provided a measurement of the quality of the content.

From the overall assessment offered by Group A it was seen that the Web based paper (average mark of 6 out of 10) was generally seen to be of lesser quality than the print based paper (8 out of 10). This is in agreement with the overall assessment of the papers by Group B, where the Web based paper was invariably assessed to be of lesser quality (with an average of 6 compared to 7 out of 10), in spite of respondents thinking that the paper was from a print based journal.

The Web based paper was also generally seen to be inferior to the print based paper with respect to accuracy, breadth, clarity, depth, and enterprise. It should be noted that although the mean assessments of these categories were invariably lower than for those for a print based paper, there is some variability both amongst respondents and between categories for any given respondent. This implies the need for caution when drawing conclusions about the overall trends in the assessments of these categories.

General comments were made by the respondents about the quality of the two papers. The Web based paper was said to be interesting but lacked conclusive results and was vague and failed to clearly explain concepts or supply sufficient information for understanding. One respondent went as far to say that they would not expect to see the paper in a print based journal. The print based paper was said to be clearer and more useful. Respondents said that the 'print' (i.e. Web) based paper was hard to follow, had too many flaws in the method and many qualifications of the results. The 'Web' (i.e. print) based paper was said to be interesting and worth pursuing with a larger, more in depth study.

In summary, the Web based paper was seen to be of inferior quality than the print based paper regardless of the attributed source. It can be concluded that the Web based paper selected was indeed of inferior quality.

Qualitative assessment of potential bias

The assessments of the Web based paper by Group A relative to the 'print' based paper by Group B provides a measure of the potential bias of respondents towards Web based papers. The overall quality of the Web based paper was marked lower when it was known to be from the Web. A higher mark was given to the paper when it was thought to be from a print based journal. This was seen by a negative difference between the Web and 'print' assessments of 0.25 out of 10. Depth and Enterprise were also marked lower when the paper was known to be from the Web. Accuracy and clarity were marked higher when the paper was known to be from the Web, and Breadth was marked equally regardless of the source.

The assessments of the print based paper by Group A relative to the 'Web' based paper by Group B provides a measure the potential bias of respondents towards print based papers. The overall quality of the print based paper was marked higher when it was known to be from a print source. A lower mark was given to the paper when it was thought to be from the Web. This was seen by the positive difference between the print and 'Web' assessments of 1 out of 10. Breadth, depth and enterprise were also marked higher when the paper was known to be from a print based journal. Accuracy and clarity were marked lower when the paper was known to be from a print source.

In summary, it appears that a negative bias exists towards Web based academic papers and a positive bias exists towards print based academic papers. In other words, a paper is seen to be of poorer quality, regardless of content, if it appears on the Web relative to if it appeared in a print based journal. This quantitative assessment agrees with the results found in the qualitative survey and past research.

It must be stressed that these results are tentative due to the very small sample size. Larger groups would be needed for the caveats to be removed. It was assumed that the groups were made up of similar types of people. This assumption was considered to be largely valid considering the careful selection of potential respondents, however larger groups of GPs and academics would be required to fully justify this assumption. It should also be noted that it was not possible to pilot the second survey. This would be essential to ensure the use of two appropriate papers in the first place.

Conclusions

The Web is a powerful communication tool, and has the potential to be a convenient survey tool. There is optimism and enthusiasm for epublishing amongst Net users and the academic community, however it is possible that a negative bias exists towards academic papers currently published on the Web. Specifically, a qualitative survey found that academics consider there to be fewer high quality publications on the Web, and it is thought that those publications that do exist appear to be of inferior quality than print based ones. The most common reason for a bias against Web based journals was a passive one; due to an academic's audience having a bias against, or not reading, ejournals, they did not consider it worthwhile to publish their best work on the Web.

Only limited conclusions can be drawn from the qualitative survey, as although the number of responses was relatively high (67) the response rate was low and highly self selected. Therefore the conclusion that there appears to be a bias against Web publications can be only tentative.

With respect to the quantitative survey, even though the sample size was small, the analysis of the results does favour a bias against Web publications. A Web paper was rated higher when thought to be from print, and a print paper was rated lower when thought to be from the Web. This agrees with the findings from the qualitative survey, past studies and the hypothesis that there is a negative bias towards Web based publications.

These findings should be built on in a future study some years hence, following the further development and use of Web based academic journals.

Acknowledgments

Thanks are due to Ilze Groves and Judy Slee for making invaluable suggestions during the design and completion of the project.

References

Bennett, J., and Aquila, R. (1995). Mental Health Program Evaluation: Effectiveness in Reducing Hospitalizations and Homelessness. Psychiatry On-Line, 1, Issue 6, Paper 2. http://www.cityscape.co.uk/users/ad88/commun.htm

Caton, C., Wyatt, R., Grunberg, J., and Felix, A. (1990). An Evaluation of a Mental Health Program for Homeless Men. American Journal of Psychiatry, 147, 286-289.

Hall, R. (1995). WWWDEV networked discussion group correspondence. Send the message 'subscribe wwwdev' to WWDEV@LISTSERV.UNB.CA.

Harnad, S. (1995). Implementing Peer Review on the Net: Scientific Quality Control in Scholarly Electronic Journals. ftp://cogsci.ecs.soton.ac.uk/pub/harnad/Harnad/harnad95.peer.review

Keown, D. (1995). Personal communication via email (d.keown@gold.ac.uk) regarding Hyperjournal-forum@mailbase.ac.uk.

Knox, E. (1993). Medical directory for Australia, 20th edition. Australasian Medical Publishing Company Ltd.

Linacre, E. (1992). Climate, Data and Resources. Routledge, 1992. 365pp.

Odlyzko, A.M. (1995) Tragic loss or good riddance? The impending demise of traditional scholarly journals. International Journal of Human-Computer Studies (formerly International Journal of Man-Machine Studies), 42, 71-122. Condensed version in Notices of the American Mathematical Society, 42 (Jan. 1995), 49-53. Available at ftp://netlib.att.com/netlib/att/math/odlyzko/tragic.loss.Z

Schauder, D. (1994). Electronic publishing of professional articles: attitudes of academics and implications for the scholarly communication industry. Journal of the American Society for Information Science, 45, 73-100.

Trevitt, A.C.F., Åkerlind, G. S. Brack C., and Pettigrove M. (1995) The role of student assessment: to gauge students' performance, to enhance learning, to develop professional skills, or to inform program evaluation ? In: P. Little, M. Ostwald, G. Ryan (eds) Research and Development in Problem-based Learning (vol 3), PROBLARC, Charles Sturt University press. pp 435-447.

Torok, S.J. (1995). Survey into opinions of academic publishing on the World Wide Web. survey.html

Williams, A., and Appelquist, D. (1995). Scientific, Technical, Medical (Peer Reviewed). http://www.edoc.com/jrl-bin/wilma/spr