Nagarajan Gopalan
5902 Aqua Bay Drive
Columbus, OH 43235
+1 614 538 1046
ngopalan@cgrg.ohio-state.edu
ABSTRACT
This paper describes the on-line testing of a prototype of an information resource for international business accessible through the World Wide Web. This study was carried out as a Masters thesis project in the Department of Industrial Design at The Ohio State University. A prototype was built and potential users were reached by identifying appropriate Internet mailing lists and posting messages to these. Feedback on the concept, content and interface was obtained through questionnaires which the users filled out before and after visiting the prototype. The methodology adopted for the study is described and advantages and limitations of this method of testing are discussed.
Keywords
User testing, On-line, Internet, World Wide Web
INTRODUCTION
A commonly asked question is "What is the Internet?".
The reason this question is asked so often is because there is no single agreed upon answer. The Internet can be thought about in relation to its common protocols, as a physical collection of routers and circuits, as a set of shared resources, or even as an attitude about inter-connecting and inter-communication. [1]
There are two ways of looking at the Internet:
Internet as Network
In technical terms, the best way of describing the Internet probably is as a global network of networks allowing computers of all kinds to directly and transparently communicate and share services.
Internet as Community
The above definition does not consider the fact that the Internet is more than just the computers which it comprises. It is also the people who use these computer networks.
Thus, at one level, the Internet is a vast collection of large and small inter-connected computer networks extending all the way around the world. At another level, it is all the people whose active participation makes the Internet a valuable information resource. [2] This paper adopts the latter view and documents an attempt at using the "Internet as Community" as a resource for user testing of a World Wide Web site.
STATEMENT OF CONTEXT
Globalization has created a highly competitive business environment where information is becoming increasingly important in every kind of business but is particularly important in today's highly complex and dynamic world of international business.
For country selection, many companies in international business first employ some form of scanning technique, looking at diverse kinds of information including market size, geography, language, market similarities, degree of red tape, costs, resource availability, barriers to entry, foreign exchange mechanisms, competitive monetary and political risk, to name but a few.
Information of this diverse nature typically needs to be obtained from a variety of sources. The time cost of acquiring information of this kind can be very high, and there seems to be a need for a resource to deliver authentic and accurate information specifically aimed at the needs of decision makers involved in international business.
The proposed solution, called WorldInfo, is the concept of an on-line information resource accessible through the World Wide Web. It is intended to be a commercial service which will generate its revenue from payments by users for the information it provides.
It addresses some of the problems with conventional on-line information sources which include requirement of membership, difficult user interfaces, high cost of membership and charges based on connect time.
The medium of the World Wide Web was chosen as a means for information delivery because of the rapidly growing reach and increasing universality of access to the Internet.
A prototype of WorldInfo was built at [3]. It resides on the Web server at the Advanced Computing Center for the Arts and Design (ACCAD) at The Ohio State University.
The following sections describe the methodology used for user testing of the WorldInfo prototype. This study was carried out as part of the author's Masters thesis project in the Department of Industrial Design at The Ohio State University.
BUILDING THE PROTOTYPE
Two prototyping options were initially considered. The first was to create a mock-up of the interface using a multi-media authoring program. The second was to actually build the prototype on the World Wide Web.
The first approach was discarded because of several limitations:
The second approach was therefore chosen as a means of achieving a practical and realistic user experience that could be tested using a diverse user sample.
Due to limitations of time and resources, it was not possible to build a fully functional prototype. The overall objective was to provide the maximum functionality possible, the objective being to give users a realistic sense of the content, structure and interaction.
Actual content was provided only in one section but each category of the information classification had a description of the kind of information that would be provided in that area. To provide a good sense of the user interaction, actual download capabilities were implemented for the information provided. Although the primary focus was on functionality, substantial effort was also put into the visual design aspects in order to enhance the realism of the user experience.
Shown below is the main screen of the WorldInfo prototype.
Figure 1: Prototype Main Screen
OBJECTIVES OF USER TESTING
Two broad objectives were identified for user testing of the WorldInfo prototype:
Usability
Usability deals largely with interface issues. How the interface is structured, how easy it is to navigate, how long documents take to load, is it possible to navigate without graphics etc.
Usefulness
Usefulness deals largely with content issues. What kind of information is provided, how is the content structured, what kinds of functionality are provided, are the documents formatted for readability/legibility etc.
Of course, usability and usefulness are both part of the overall user experience and it is not entirely possible to separate these two aspects. However, there is an essential difference between the two:
A third testing objective was to get feedback on the overall feasibility of the concept. There were two sub-objectives in this area:
Confirmation of Need
The WorldInfo concept was based on assumptions about the problems with existing conventional and on-line sources of information. It was felt necessary to confirm that these problems really exist.
Commercial Feasibility
Most of the Web based information sources presently are based on the sponsorship model- the users do not pay for services and the cost is borne by advertising revenue. It was therefore necessary to determine whether users would accept a commercial pay-per-use service such as WorldInfo.
LOCATING THE SUBJECTS
The Internet supports a large, global on-line community and therefore has the potential for use to reach a large audience. But, in order to utilize it effectively for user testing in any form it is necessary to be able to target specific user groups with clearly defined profiles.
This may seem to be a daunting task considering the size and distributed nature of the Internet. Fortunately, this task is made easier by the fact that various communities of interest already exist in the form of Mailing Lists and Newsgroups. These provide a means of reaching specific groups of users.
USENET Newsgroups
USENET Newsgroups are the Internet's equivalent of a discussion group or a BBS (Bulletin Board System). Essentially, Newsgroups are on-line communities of shared interests. The subject of these shared interests are extremely diverse and it can be extremely difficult to identify the right Newsgroups for a specific purpose.
Internet Mailing Lists
An Internet mailing list is a community of common interests, similar in some ways to Newsgroups. The essential difference is that Mailing Lists rely on Email as a means of delivery of information and messages. Like Newsgroups, the subjects of various mailing lists are extremely diverse and the most difficult task is to identify the right mailing list for one's purposes. Mailing lists have several advantages over Newsgroups. They deliver messages directly without requiring any special effort on the part of the user (beyond retrieving their Email, and in many situations particularly in the work environment, systems are setup so as not to require any special action on the part of the user). Also Email is more or less ubiquitous on the Internet whereas access to USENET is not. Also, because of the way USENET News propagates, messages from mailing lists often arrive much faster.
For these reasons, it was decided to use Mailing Lists rather than Newsgroups to reach the target user group.
REACHING THE SUBJECTS
There are thousands of Mailing Lists dealing with a diverse range of subjects. Identifying the right Mailing Lists can be a very difficult and challenging task.
Fortunately, there are means available to do this in the form of indices (some of which are searchable) of Mailing Lists which can be accessed through the World Wide Web.
Some of these are:
Interlinks at Nova Southeastern University
http://www.nova.edu/Inter-Links/cgi-bin/lists
Liszt of Mailing Lists
http://www.liszt.com/
Stephanie da Silva's List of Mailing Lists
http://www.neosoft.com/internet/paml/index.html
T I L E . N E T / L I S T S
http://www.tile.net/tile/listserv/index.html
Considering the nature of the project and the objectives of testing, it was felt that feedback should be obtained from two classes of subjects:
Searches using the indices listed above yielded a total of 25 mailing lists which appeared (from their names and limited information available from the indices) to be appropriate for the purposes of the study.
The next step was to confirm the appropriateness of these mailing lists for the purpose of the study. This was necessary not only from the point of view of the authenticity of the results but also to avoid "flaming". Mailing Lists and Newsgroups have their own set of rules of behavior, usually referred to as Netiquette. These communities can be very intolerant and often, inappropriate postings to mailing lists can result in one receiving many angry responses from all over the world (referred to as "flaming").
The best way of ensuring appropriateness is to first subscribe to a mailing list and "lurk" in the background: observing the messages being posted in order to get a clear idea of the people on the list and their interests. Another way of determining appropriateness is to download a list of names and Email addresses of members. Looking at the domain names of users usually gives an idea of the nature of people on the list.
By doing this, the total number of lists was reduced to 15: several lists turned out to be inactive while others turned out to be dealing with issues other than that suggested by their names. The total number of subscribers to these lists exceeded 6000 people and the sizes of the individual lists varied widely, ranging from 79 to 1675 members.
Messages were posted to these lists provided information about the project and its objectives and inviting those interested to visit the prototype and provide feedback. This was done after pilot testing with a limited number of local subjects to ensure that the instructions provided were clear and understandable.
RESEARCH INSTRUMENT
Two questionnaires were provided in the prototype for the purposes of obtaining feedback. The first was an Entry Questionnaire to be filled in before visiting the prototype and the second an Exit Questionnaire to be filled in after visiting the prototype.
The nature of information sought in these two questionnaires was distinctly different. The first was aimed primarily at understanding the nature of user needs and the problems with existing sources of information: these questions would best be asked without biasing the subjects by exposing them to a proposed solution. The focus of the second questionnaire was on the solution itself and necessarily had to be filled in after visiting the prototype. Another reason for having two questionnaires was to avoid overloading the user by requiring them to provide too much information in one session. The URL given in the messages posted to the Mailing Lists was not that of the prototype itself but took users to a page with a brief background statement including a description of the objectives and features of WorldInfo. This page then led to a page with instructions which then led to the Entry Questionnaire.
There was no way of forcing people to proceed to the Exit Questionnaire after visiting the prototype. Many people did not get to the Exit questionnaire though the instructions that visitors received prior to the entering the site had bold and specific reference to this. A reminder was also provided through a script included in the entry page of the prototype that caused a message to scroll through the bottom of the browser window. Additionally, a reminder Email message was sent to those who provided an Email address in the Entry Questionnaire.
Entry Questionnaire
The Entry Questionnaire aimed at getting background information about the subjects and their information needs and sources. Information requested included Contact Information (Optional), Professional Background, Access to the Internet, Usage of Internet Information Resources, Information Needs and Sources, Satisfaction with Conventional and On-line Information Sources. Most of the items in the Entry questionnaire involved selecting items from pop-up menus or keying responses into text boxes, though there were several open ended questions too.
Figure 2: Entry Questionnaire
On submitting the Entry Questionnaire, the users were taken to the prototype from where access to the Exit Questionnaire was provided through an "Exit" button which was provided in the navigation bar of the prototype.
Exit Questionnaire
The Exit Questionnaire aimed at getting feedback on the site itself and suggestions for improvement. It was kept very short since it was felt that people would not have the patience to fill out a long questionnaire after spending time visiting the prototype.
The Exit Questionnaire had just 6 items. Five of these items required ratings of the prototype in terms of Content, Usefulness, Organization/Structure, Navigation/Ease of Interaction and Overall Commercial Feasibility. A 7 point scale (-3 to +3) was used and respondents were requested to rate the prototype on these parameters by agreeing or disagreeing with a statement following each parameter. The sixth item was an open-ended question which requested comments and suggestions for improvement.
The Exit Questionnaire also provided a set of links to the major sections of the prototype in order to make it easy for users to refresh their memory about any specific aspect, if they felt that to be necessary.
Figure 3: Exit Questionnaire
RESULTS
Responses to the Mailing List postings were practically instantaneous with the first Entry Questionnaire being received within 30 minutes. though the response rate dropped off rapidly thereafter as shown below.
 |
Figure 4: Questionnaire Responses
Responses continued to trickle in even 2 weeks after the posting but the findings presented below are based on the Questionnaires received in the first week. There were 91 Entry Questionnaires during that period whereas only 41 Exit Questionnaire responses were received.
The questionnaires were processed by a script that converted the inputs to the questionnaire into an Email message. Submission of a questionnaire automatically took the user to the next document: in the case of the Entry Questionnaire, this was the main screen of the prototype; in the case of the Exit Questionnaire, this was a message of thanks.
The script was set up in such a way that it returned information about the operating platform and the exact version of browser being used as well as the IP (Internet Protocol) address from which the questionnaire was submitted. This was done for the purpose of allowing tallying of the Entry Questionnaires with the corresponding Exit Questionnaires.
Entry Questionnaire
Feedback from the Entry Questionnaire largely tended to confirm the information needs and problems with existing sources which formed the basis of the proposed solution. Some of the key findings were:
Exit Questionnaire
As mentioned earlier, there was no way of ensuring that the subjects did indeed proceed to the Exit Questionnaire after visiting the prototype. Consequently, the number of responses received was significantly less: the findings presented below are based on a total of 41 responses (compared to 91 Entry Questionnaires).
Ratings of content and usefulness in the Exit Questionnaire as well as comments received indicated that the information structure and scheme of the prototype worked well and was appropriate for the intended user group. Likewise, the feedback on organization and navigation indicated that the WorldInfo prototype is fairly effective in this area too.
The average and standard deviation of responses (converted to a scale of 1 to 7) are given below.
| Mean | S.D. | |
| Content | 5.10 | 1.39 |
| Usefulness | 5.10 | 1.49 |
| Organization | 5.22 | 1.47 |
| Navigation | 5.15 | 1.70 |
| Feasibility | 4.63 | 1.66 |
Figure 5: WorldInfo Ratings
Overall, comments and suggestions in the open-ended part of the Exit Questionnaire were very positive. The key issues raised were:
ADVANTAGES AND LIMITATIONS
The major advantages of the methodology adopted in this study include:
Quick Results
This method has the benefit of extremely quick response times. As indicated earlier, responses to the posting messages were practically instantaneous: The first response was received within 30 minutes of posting and 60 per cent of responses considered were received within 48 hours. Responses tapered off rapidly after that although some were still coming in even two weeks after the initial posting.
Ease of Access to Users
Using this methodology, it is possible to easily reach a large group of potential subjects from all over the world just by sending out a few Email messages. For example, in the present case, it was possible to reach over 6,000 potential subjects by sending out just 15 Email messages.
Access to Diverse User Groups
The testing objectives of the present study required reaching an audience with expertise or involvement in areas as diverse as International Business, Industrial Design, Information Design, Electronic Commerce and Management Information Systems. Locating and reaching such diverse user groups would have been extremely difficult through any other means.
Realistic User Experience
The methodology adopted in this study allowed a fairly realistic simulation of the actual on-line interactive user experience, better than would have been possible with user testing in a laboratory setting.
However, this methodology also suffers from some limitations including:
Academic Audience
The methodology adopted in this study allows reaching a primarily academic audience. Most Internet Mailing Lists allow subscribers to download a list of Email addresses of all the subscribers to the list. This was done in the present case to try and get an idea of the appropriateness of the Mailing List for the purposes of the study. Inspection of these lists revealed that the majority of people on the Mailing Lists identified had Email addresses corresponding to academic domains. While this could be limitation in some contexts, it could actually be a benefit if the objective is to get inputs from expert in specific fields.
Limited Opportunity for Observation
This methodology does not allow actual observation of user's action, the path they use for navigation, the problems they are having etc. This could be achieved through in-laboratory testing but could also be achieved in on-line testing by recording server access requests or analyzing Web server access logs. This could not be done effectively in the present study due to limitations of server access though a limited attempt at indirect observation was made by providing access counters recording the number of hits received by key documents within the prototype.
Motivation of Subjects
One of the limitations of the methodology described is that subjects were not getting anything specific in return; they were just responding to a student's request for help. Consequently, their level of involvement and commitment may not have been as high as desired. Also, since subjects are not getting anything out of participating, they are unlikely to respond again if one were to go out again with a new version for testing; it is therefore extremely critical to get it right the first time. However, it should be possible to overcome this limitation by offering participants something concrete in return for participation.
High Implementation Level
For this method to be effective a comparatively high level of functionality needs to be built into the prototype. This is particularly so because there are limited opportunities for interacting with subjects or explaining to them how things would actually work. Consequently, this method is probably more suitable for confirmatory rather than exploratory research.
CONCLUSION
The methodology described in this paper offers a powerful means of quickly and easily reaching a wide and diverse population of subjects.
It is particularly suitable for reaching a predominantly academic audience. While this could be useful when it is desired to reach an audience of experts in specific fields it may, in some ways, limit its applicability in commercial contexts.
Compared to laboratory testing, it has the advantage of providing a more realistic and contextual user interaction experience. The trade-off is in the limited possibilities for direct interaction with or observation of users.
A comparatively high level of functionality needs to be incorporated in the prototype for this method to be effective. This makes it more suitable for confirmatory rather than exploratory testing. The biggest limitation of this methodology is probably in terms of the motivation and involvement of the subjects. This can be overcome by providing some form of concrete return to the subjects in return for their participation.
REFERENCES
1. Krol, Ed. The Whole Internet User's Guide and Catalog. Sebastopol: O'Reilly & Associates, 1994.
2. Butler, Mark. How to Use the Internet. Emeryville: Ziff-Davis Press, 1994.
3. The WorldInfo Prototype. Available at http://www.cgrg.ohio-state.edu/~ngopalan/thesis.html