Article Review: Unexpected Benefits of Return on Investment Analysis

This article was originally reviewed for my statistics and research methods class.

Pan, D., Wiersma, G., Williams, L., & Fong, Y. S. (2013). More than a number: unexpected benefits of return on investment analysis. Journal of Academic Librarianship, 39(6), 566-572.

Since the crash of 2008, libraries (along with everyone else) have struggled to provide more resources with less funding, frequently under close scrutiny from administrators and other higher-ups who want to make sure that the money being spent is having a measurable effect on the library’s community. Because of this, libraries have had to develop new metrics for measuring their impact and value, usually in the form of cost-benefit analyses (CBAs) or return on investment (ROI) analyses. Unfortunately, measuring the impact of a library isn’t the easiest thing; this study from the University of Colorado aims to develop a methodology whereby librarians can measure the relationship between library resources and scholarly output using both qualitative and quantitative methods in order to clearly show that library expenses are “worth it.”

This study immediately reminded me of another article I read this semester, “Overlap between humanities faculty citation and library monograph collections, 2004-2009” (Kellsey & Knievel, 2012) [1], since the study seemed like it was going to be similar. However, one of my issues with the Kellsey article was that the researchers had used citation analysis to determine faculty use of collections without actually verifying that the faculty had used library collections. Pan et. al not only analyzed citations, but also took the step of interviewing faculty members to discuss their perceptions of library services and resources. Their method of combining quantitative usage statistics with qualitative “explicit values” (p. 568) seems more thorough than mere quantitative analysis.

However, there were a few things about their methodology that concerned me, particularly the sampling methods. The authors mention that time was a factor, but it seemed odd to me that they utilized only faculty members who do research, and asked for syllabi as well as curriculum vitae. If they were going to ask for syllabi, why not ask teaching faculty to particupate as well? Younger/untenured faculty are just as likely (if not moreso) to use the library, particulary since they don’t generally have funding of their own and depend on library materials. In addition, these teaching faculty are going to be the ones pointing their students towards the library – and students use resources as well. I also found it odd that two of the participating campuses focused on a single department, while the Boulder campus recruited participants from across multiple departments. Perhaps this makes sense in terms of the way the institution is structured, but from the outside, these just seemed like odd sampling choices.

The researchers’ choice to use their study as a chance for librarians to engage with faculty about library services was also an interesting one, and while I found the descriptions of the interviews to be a slight tangent from the main thrust of the article, I felt that the method they described sounded like a useful one that could be incorporated into other libraries’ assessments. Additionally, their clear explanation of their calculations was helpful for any who might consider using their method without being overwhelming.

While the authors of this study do caution against drawing any definite conclusions from their research because of the small sample size, their methodology does seem relatively sound. If I were to use the method for my own research, I might consider including a more varied sample size, but overall, I think this pilot study was sucessful and that the method could be generalized to other institutions who might be looking for a more comprehensive measure of value than simple usage statistics. Both librarians and administrators might find this study to be a helpful example that could assist with their assessments.

1: Kellsey, C. & Knievel, J. (2012). Overlap between humanities faculty citation and library monograph collections, 2004-2009. College & Research Libraries, 73(6), 569-583.