The overall methodology used for testing involved having potential users and others with relevant expertise actually visit the prototype and provide feedback. The audience for this exercise was reached mainly by posting information about WorldInfo to relevant mailing lists. Feedback was obtained through on-line questionnaires. The details of the study and the results are described in the following sections.

9.1 Reaching the Audience

It was felt that feedback on the WorldInfo concept should be obtained not only from potential users but also from people with expertise in International Business, Industrial Design, Information Design, Electronic Commerce and Management Information Systems.

Since feedback was to be obtained by actually having the subjects visit the prototype, the subjects were contacted through mailing lists dealing with these subjects: he audience reached by this method would be more likely to have full Internet access. Of course, this method had the limitation that it would reach a predominantly academic audience. But this was also an advantage because this audience was more likely to have the time and inclination to visit the prototype and provide feedback.

Mailing lists were identified by running searches on the following indexes of mailing lists:

Interlinks at Nova Southeastern University

http://www.nova.edu/Inter-Links/cgi-bin/lists

International Federation of Library Associations and Institutions

http://www.nlc-bnc.ca/ifla/I/training/listserv/lists.htm

Liszt of Mailing Lists

http://www.liszt.com/

Mailing List Archive at Indiana University

http://scwww.ucs.indiana.edu/mlarchive

Stephanie da Silva's List of Publicly Accessible Mailing Lists

http://www.neosoft.com/internet/paml/index.html

T I L E . N E T / L I S T S

http://www.tile.net/tile/listserv/index.html

These searches yielded a total of 25 mailing lists. In order to assess the appropriateness of these mailing lists for the purposes of this study, the author subscribed to all of them and observed the nature of traffic on them. Several of them turned out to be inactive while others turned out to be dealing with issues other than that suggested by their names.

This reduced the number of lists to the following 13. Lists of subscribers for these were downloaded (wherever possible) in order to get an idea of the size of the audience for the posting messages and the figures are noted against each.

The total number of subscribers to these lists exceeded 6000 people. A look at the lists of Email addresses confirmed that, as expected, a large portion of them had addresses corresponding to educational domains. The sizes of these mailing lists varied widely, ranging from 79 to 1675 members.

In addition to the above mailing lists, the MBA students at The Ohio State University, were targeted using the College of Business Email reflector. About 500 members of the Industrial Design Society of America (IDSA) who had Email addresses listed in the 1996 IDSA Directory were also sent information about the prototype.

The following is the text of the message was posted to the above mailing lists on April 21, 1996:

9.2 Testing and Feedback Mechanism

Two questionnaires were provided in the prototype for the purposes of obtaining feedback. One was an Entry Questionnaire that was to be filled in before visiting the prototype and the other an Exit Questionnaire to be filled in after visiting the prototype.

The URL given in the posting messages was not that of the prototype itself but was that of a page with a brief description of the objectives and features of WorldInfo. This page led to a page with instructions which then led to the Entry Questionnaire. On submitting the Entry Questionnaire, the users were taken to the prototype automatically.

Entry Questionnaire

The Entry Questionnaire aimed at getting background information about the visitors as well as information about their information needs and sources. Most of the items in the questionnaire involved selecting items from pop-up menus, though there were several open ended questions too. The Entry Questionnaire had the following sections:

Personal Information (Optional)

Name, Email, Phone Number and Address.

Background

Profession, Age, Sex, Annual Income

Access to the Internet

Mode of Access, Speed of Access, Experience of the Internet, On-line time per week

Usage of Internet Information Resources

Frequency, Method of Locating Resources and Archiving Method

Information Needs

An open ended question requesting information about needs.

Information Sources

An open ended question requesting information about sources.

Satisfaction with Conventional Information Sources

An open ended question requesting information about problems with conventional sources.

Satisfaction with On-line Information Sources

An open ended question requesting information about problems with on-line sources.

Other Information

An open ended question requesting any additional relevant information

Exit Questionnaire

The Exit Questionnaire aimed at getting feedback on the site itself and suggestions for improvement. This was kept much shorter since it was felt that people would not have the patience to fill out a long questionnaire after spending time visiting the prototype. Access to the Exit Questionnaire was provided through an "Exit" button which was provided in the navigation bar. The instructions that visitors received prior to the entering the site had specific reference to this but there was no way of forcing people to proceed to the Exit Questionnaire after visiting the prototype. A prominent visual of the button was used in the hope that this would help visitors to remember this requirement. An additional reminder was provided through a script included in the entry page that caused a reminder message to scroll through the bottom of the browser window.

The Exit Questionnaire had just 6 items- 5 of which required ratings on a scale and 1 which was an open-ended question which requested comments and suggestions for improvement.

Respondents were requested to rate the prototype on a 7 point scale (-3 to +3) for the following parameters by agreeing or disagreeing with the statement following each parameter:

Content

The proposed content is appropriate and adequate for the intended user group.

Usefulness

The information and service proposed to be provided by WorldInfo would be useful to the intended user group.

Organization/Structure

The organization and structure of the system was logical and clear.

Navigation/Ease of Interaction

The site was user friendly and easy to navigate.

Overall Commercial Feasibility

The concept behind this site is commercially feasible and the intended users would be willing to pay for the convenience, ease of use, reliability of information and rights of usage that it will provide.

The questionnaires were processed by a script that converted the inputs to the questionnaire into an Email message. The script was set up in such a way that it also returned information about the exact version of browser being used, the operating platform and the IP address of the respondent. This was primarily set up for the purpose of allowing tallying of the Entry Questionnaires with the corresponding Exit Questionnaires.

9.3 Responses and Results

A total of 91 Entry Questionnaire responses and 41 Exit Questionnaire responses were received as of May 12, 1996.

Responses to the postings were immediate but the response rate fell off very rapidly. 38 Entry Questionnaires were received within the first 24 hours, 55 within 48 hours and 62 within 72 hours. Surprisingly, occasional responses were still coming in even 2 weeks after the first posting.

Many respondents to the Entry Questionnaire did not fill out the Exit Questionnaire. Reminders were sent out to the people who had provided their Email addresses but this was not successful in eliciting responses.

Entry Questionnaire (91 responses)

Some of the key findings from the Entry Questionnaire responses are:

• Very few respondents were using UNIX computers (11%). The majority (54%) were using PCs while the remainder (35%) were using Macintoshes.
• A majority of respondents (74%) used Search Engines for resource discovery. This is probably a reflection of the fact that presently, there is no other effective means of finding information on the Internet.
• 52% of respondents reported that they archived information from the Internet by saving it to disk. This suggests that automatic downloading of files in a user specified format (as envisaged in WorldInfo) could be of value to users.
• Problems cited with conventional sources of information include currentness/accuracy (9 responses), difficulty in physically getting material (6), cost (7) and non-electronic format (3).
• The most frequently indicated problem (35 responses) with on-line sources (which most respondents interpreted as meaning Internet sources) was the fact that it is difficult and time consuming to find information. This would tend to indicate that there is potential for a resource that meets all the information needs at a single location and thus eliminates the need for searching.
• Doubtful quality of information available on the Internet was cited as a problem by 15 respondents, indicating that reliability and authenticity of information could be an important selling point.

Exit Questionnaire (41 responses)

The average and standard deviation of responses (converted to a scale of 1 to 7) are given below:

Suggestions/Comments

Overall, comments and ratings (above) of the prototype were very positive. The key issues raised were:

• Several respondents felt that the commercial feasibility was doubtful because there is a lot of free information available on the Internet. Advertising/sponsorship was suggested as the route for generating revenue.
• Others felt that the commercial feasibility would depend largely on the quality of information provided and the pricing structure.
• A number of respondents felt that a lot more actual content needs to be provided in order to get meaningful feedback on the feasibility of the concept.
• A few respondents suggested the need to reduce the number of levels the user has to traverse before getting to actual information.