Blue text are a review of this "expedited cursory review" by Diane A. Burton. It is of note that USE took the concepts of "expedited" and "cursory" to the extreme in the conduct of its reveiw of the 260 page Boyd County Sosiocultural Assessment (BCSA).
Purpose of Review
This review was prepared by US Ecology at the request of the Central Interstate Low-Level Radioactive Waste Compact Commission. The Commission's instruction was to expedite a cursory review of the Sociocultural Assessment so that the Commission could respond in a timely manner to a request from the Boyd County Local Monitoring Committee. [Emphasis supplied.]
To accomplish this goal, US Ecology was instructed by Commission staff to limit the review to 1) a general assessment of validity of the report's major conclusions and; 2) identification of recommendations contained in the report.
Method of Review
US Ecology first conducted a general overview of the Sociocultural Assessment Final Report to determine how extensive a review would be necessary to respond to the Commission's request.
The general overview found that the study had self-acknowledged limitations. These limitations, which have not to our knowledge been publicly discussed, make it difficult to substantiate the overall validity of summary statements. Because of this, US Ecology considers it inappropriate to discuss the relevancy or implications of summary statements. The objectives of the Commission can be accomplished without such discussion.
To achieve the Commission's objectives, this review focuses on specific portions of those report sections that set forth the scope of work and the authors' conclusions and recommendations. Those sections are: 1) Data Limitations; 2) Methods and Procedures; and 3) Conclusions and Recommendations.
[Note: Data Limitations is not a separate section, it is a part of the Methods and Procedures section.]If, after studying US Ecology's cursory review, the Commission considers it necessary to more fully examine the document, we recommend that a comprehensive analysis be conducted by an impartial third party.
1
In the section Data Limitations, the authors state that the scope of their assessment was too limited to determine with any certainty to what extent conditions that existed prior to selection of the Butte site contributed to current county sentiment. The Assessment states:
"Most importantly, the researchers did not have access to true baseline data on local social and cultural conditions prior to (the) time when the LLNW facility became a public issue... Although the research reported here can document many changes which have taken place, the absence of a true pre-project baseline limits the potential for establishing the magnitude of such things or the relationships between apparent change and the proposed LLNW facility." (Pgs. 20-21)
The above quoted section of the BCSA are sentences from two separate paragraphs. The full paragraph for the first sentence quoted is:
"However, even with these overwhelming advantages there remain important data limitations which constrain the types of conclusions that can be drawn from this research. Most importantly, the researchers did not have access to true baseline data on local social and cultural conditions prior to (the) time when the LLNW facility became a public issue. Instead, the research was conducted more than three years after the possibility of siting the facility in Boyd County was first announced in 1988, the time when an ideal SIA should have been initiated."USE's failure and refusal to conduct a sociocultural assessment during the site characterization phase contributes to the lack of true baseline data. Interestingly, USE did contract for a telephone survey in all three candidate counties in March of 1989. That telephone survey asked multiple questions about the respondent's support for a LLRW disposal site, as well as other attitudinal questions. To this date (November, 1997), USE has failed and refused to release the results of that March, 1989, survey. While by no means a comprehensive SIA, the results of that survey would provide at least minimal baseline data.
The authors also acknowledge that the scope of work severely limited their ability to predict with any certainty future events in Boyd County. The Assessment states:
Similarly, efforts to predict future impacts of the project are constrained by the need to rely on data collected only during a relatively brief research period prior to actual licensing, construction, or facility operation. Under the best circumstances, multiple over-time measures would have been made, allowing for more valid predictions of future behavior. Synchronic data are simply inadequate for trend analysis. Although such limitations are not uncommon.. .they are compounded in this case by the limited information currently available. ..(e.g., the actual quantity and types of waste...waste transportation routes, transportation schedules, housing plans for construction and operations workers, etc.)" (Pg. 21).
Once again, USE chose to selectively quote from a paragraph. A key phrase from the above quote was omitted, that phrase came after the word 'available' in the final sentence of the quote. The missing phrase is, "...regarding some aspects of the proposed facility and its operation..." The lack of information in this instance can be directly traced to USE's failure to have provided such imformation in its license application.
In the section Methods and Procedures, the authors acknowledge the assessment was further limited by financial constraints. They state:
"Funding restrictions made a full ethnographic description of the entire county impossible. so study efforts concentrated on areas nearest the proposed site. It was assumed that residents living near the proposed site would likely be more effected by the project and would therefore be more involved in and aware of the issues than those living further from the site" (Pg. 16).
It is more than reasonable to assume that those living closer to the proposed site would be more effected by the proposed project.
The US Ecology review also identified another potential study limitation not clearly identified as such by the Assessment authors.
"Questionnaires were personally delivered to members of the sample... The fieldworkers returned to retrieve the completed questionnaires within a day or two after they were delivered" (pg. 19).
This sampling procedure does not include sufficient controls to assure that responses given accurately reflect the attitudes of survey participants. The procedure does not preclude the possibility of discussions occurring between participants and/or with other parties interested in influencing the survey results. One can not say with certainty that such discussions did not take place, the extent to which they might have influenced individual survey results, or the extent they might have influenced general survey results and authors' conclusions.
Firstly, the authors of the BCSA did note the potential for someone other than the selected respondent completing the questionnaire -- apparently the expedited cursory review did not allow USE to discover that portion of the BCSA. And once again, key portions of the quoted paragraph are omitted, the full quote is:
"Questionnaires were personally delivered to members of the sample by two trained fieldworkers who gained the cooperation of the selected person before leaving the questionnaire. The fieldworkers returned to retrieve the completed questionnaires within a day or two after they were delivered. In order to obtain the highest degree of cooperation, an article describing the project and a photgraph of the fieldworkers were published in the county newspaper immediately before the start of the survey and during the first week of survey administration. In addition, each fieldworker possessed letters from the leadership of both PFP [People for Progress] and SBC [Save Boyd County] supporting the research endeavor and encouraging their supporters to cooperate in filling out the questionnaires."Secondly, any potential for "discussions occuring between participants and/or with other parties interested in influencing the survey results" were just as likely to occur among proponents of the LLRW project as opponents of the project. One would expect such an "influence" to occur at a rate no higher than chance.
Conclusions
The section Conclusions and Recommendations includes ideas assumed to have been drawn from information in preceding sections. Because of the limited review conducted by US Ecology, no attempt has been made to determine if any of the authors' conclusions are fully supported by information contained in their report or the scientific validity of that information.
As stated previously, the nature and scope of the Sociocultural Assessment does not provide sufficient mechanisms for determining the accuracy of its data. This is especially significant because of the reliance on opinions volunteered by Boyd County residents. Apparently the authors have elected to assume that the opinions offered through interviews and survey responses are factual accounts not affected by outside influences.
For instance, the authors do not provide any mechanism to offset the possibility that some persons interviewed might have felt that it was in their best interests to respond in the most extreme way. Further, the way in which some of the data was collected (no research personnel present during the filling out of questionnaires) introduced a variable that makes it impossible to determine who actually responded.
While a disadvantage of using self-administered questionnaires is the inability to control the specific respondents who answers the questionnaire, there also are disadvantages with other survey methods. When an interviewer is invovled, interviewer bias can occur; additionally social desirability bias is more common with telephone or face-to-face survey designs. Further, self-administered questionnaires are advantageous over telephone or face-to-face interviews, in that self-administered questionnaires garner fewer non-responses on senitive questions, and respondents give more candid answers. And finally, USE does not acknowledge that the BCSA utilized multiple data collection strategies. The use of multiple data collection strategies is known to increase both the validity and reliability of research results.
Because of this, one could easily hypothesize that respondents may have found it beneficial to discuss and agree upon the most "appropriate" answers, based upon their position regarding the LLRW project. Given the relatively small population base, such cooperation could significantly skew results. Such cooperation could also be theorized to be the cause of the high respondent rate (over 87 percent of the questionnaires were returned).
This is perhaps the weakest of USE's challenges to the BCSA. The high response rate should have been expected.
"Recent research suggests that the mail questionnaire [self-administered] and telephone interviewing are most appropriate for modern, fluid postindustrial society. In fact, it is the mail questionnaire that has held and improved on response rates, while face-to-face interview rates of response have declined." (Frey, pg. 52)"When comparing the response rates for all three methods, Siemiatycki, using regional samples, produced initial rates of 70.3 percent for mail, 73.5 percent for telephone, and 84-1 percent for face-to-face. Subsequent follow-ups to the mail and telephone surveys pushed these rates to equal or exceed that of face-to-face." (Frey, pg. 51)
"Response rates in mail surveys are extremely varied, ranging from as low as 10% to over 90%. This variation depends in part on the efforts made with follow-ups, and on the subject of the survey and its relevance to the survey population." (Kalton, pg. 66) [Empahsis supplied.]
In addition, the Sociocultural Assessment authors acknowledge other limitations that alone make it impossible to determine with any certainty the causes of current county sentiment or to predict future events. Given this, the statements offered as "conclusions" should more accurately be termed author's speculation.
Our review discovered a significant omission that must be taken into account when evaluating the accuracy of some public accounts of the document. The Conclusions and Recommendations section lacks any reference to the limitations acknowledged elsewhere by the authors.
It is our opinion that many of the public statements made regarding this report resulted from reliance on this one section as a means of evaluating the entire report. Had the authors referenced these limitations in the Conclusions and Recommendations section, it is quite likely that the public's perception of the document would be dramatically different.
Recommendations
The authors of the Sociocultural Assessment offer several recommendations for improving relations between the Central Interstate Project and people living in and near the affected area. To date, we are not aware of any public discussion regarding these suggestions.
This may be due, in part, to the fact that several pages of narrative placed at the beginning of the Recommendations section more rightly should have been placed in the preceding section. This organizational error may have inadvertently obscured the actual recommendations for ninny readers.
It is unfortunate that the CIC and USE paid no more attention to the BCSA than to conduct an "expedited cursory review" of a 260 page document. The lack of attention to areas such as this only serves to increase the lack of confidence that the people of Boyd County, and Nebraska at large, have in the CIC and USE.
References: James H. Frey, 1989. Survey Research by Telephone. Newbury Park: Sage Publications, Inc.; Graham Kalton, 1983. Inroduction to Survey Sampling. Newbury Park: Sage University Press; and Herbert F. Weisberg, et al., 1996. An Introduction to Survey Reserach, Polling, and Data Analysis. Thousand Oaks: Sage Publication, Inc.
Read the Methods and Procedures and Conclusions and Recommendations sections of the Boyd County Sociocultural Analysis for yourself.
Return to HOPE
