Jian Wang, Portland State University Library
Hannah McKelvey, Montana State University Library
The purpose of this document is to meet the charge of the SPARC Data Analysis Working Group (DAWG) and provide a high-level overview of why libraries collect online resource usage statistics, what types of usage statistics they can collect, and ways they collect this usage data.
Usage statistics in libraries
Usage statistics deal with collecting, analyzing, and presenting data (Mood, 2018). Libraries collect various statistics for planning, developing, and evaluating their services. Some examples include circulation, visits, collection, acquisitions, electronic resource usage, reference/chat transactions, and library instruction sessions. This document focuses on electronic resources usage statistics in relation to budget allocation, collection development, deselection and cancellation decisions, and negotiations with vendors. In this context, library usage statistics can be defined as a “specified set of data elements to help libraries measure the use of their electronic resources” (Baker & Read, 2008) to evaluate subscriptions and to determine value and return on investment (ROI) for libraries.
Why do we need to collect usage statistics?
Libraries collect usage statistics for a variety of reasons, from helping them understand the value of their subscription resources to requirements for high-level reporting. Below are common scenarios where usage statistics would be of use, examples of standard reports that libraries are asked to complete, and a non-exhaustive list of metrics that can be calculated with usage statistics.
Common reporting scenarios
Often, libraries are required to report usage statistics at the state and university levels. For example, Portland State University (PSU) Library has a separate, off-site storage facility which houses a significant number of books. The university’s Office of Risk Management requests that the PSU Library sends an annual storage inventory report for insurance purposes (End-of Year Insurance Evaluation Stats). If the storage facility were to “go up in flames,” the University would need as much detail as possible about what was lost in order to file an insurance claim. In addition to insurance purposes, library usage statistics can be used:
- for negotiations with publishers
- to view trends over time
- to inform collection management decision
- to measure turnaways for renewals/deselections/package swaps decisions
- to help with justifying library materials budget allocations, etc.
- to advance university enrollment, retention, and recruiting effort
Each year, libraries are asked to complete standard reports so that library data can be aggregated and shared nationally. The two most common national reports are from ACRL and IPEDS:
- Association of College and Research Libraries (ACRL) Academic Library Trends and Statistics Survey: “designed to gather information at the national levels from all types of academic libraries” (ACRL, 2020).
- ACRL Academic Library Trends and Statistics Survey
- ACRL Annual Survey Instructions and Definitions (PDF) – Revised: February 2021
- ACRL requires the following electronic usage statistics:
- Question 60 Column B: Total Digital/Electronic Circulation or Usage
- Questions 61 and 62: COUNTER Release 4 Circulation or Usage
- Question 63: E-serials Usage
- ACRL requires the following electronic usage statistics:
- Integrated Postsecondary Education Data System (IPEDS): Academic Libraries Survey: “The purpose of the Academic Libraries Survey (ALS) is to collect information on library resources, services, and expenditures from academic libraries serving degree-granting, Title IV postsecondary institutions in the 50 states, the District of Columbia, and the outlying areas” (IPEDS, n.d.).
- IPEDS Academic Libraries Information Center
- IPEDS 2020-21 Survey Materials/Form and Instructions
- IPEDS requires the following electronic usage statistics.
- Total Digital/Electronic Circulation or Usage
Usage statistics calculations and uses
Usage statistics can be used to calculate specific metrics or generate visuals that help libraries better understand and assess how their online resources are being used. Some examples include:
- Return on Investment (ROI): the income—or value—received as a result of an amount invested in an asset (Kaufman & Watstein, 2008, p. 227).
- Social Return on Investment (SROI)/Student Return on Investment (SROI): an alternative/additional measurement to capture and measure the social value of library services and resources to demonstrate the wider value (Kelly et al., 2012; Luther, 2008; Pan et al., 2013; Mezick, 2007; Watson et al., 2016; Malapela & Jager; 2018; Pan et al., 2014). Examples of value calculators include:
- UWF Libraries Value Calculator
- Madison College Libraries Value Calculator
- PALNI Student Return on Investment Calculator
- Data visualization: term used to describe all of the ways people transform data into visual representations (Duke University Libraries, n.d.).
- Cost Per Use: the total subscription cost divided by the number of times a resource was used will generate a cost-per-use figure. Issues with cost per use include failure to take into consideration variability in the nature of usage. Unsub offers an alternative data-analysis tool for libraries to analyze and estimate the cost and value of serials subscriptions (Hinchliffe, 2020; SPARC, 2020; Kendrick, 2019; Arthur, 2018; Bucknall et al., 2014; Glasser, 2018; Huffine, 2015; Hulbert et al., 2011; King & Tenopir, 2013; Smulewitz et al., 2013).
- Turnaways: usage data for turnaways can help libraries identify gaps or an “unmet need”. There are some caveats concerning turnaway counts. Please review the Turnaways article for more information about key factors to consider when reviewing vendors’ turnaway reports.
Please review the Data Analysis Glossary for further examples or metric definitions.
What types of usage statistics do we (and can we) collect?
There are several specific metrics that libraries can collect to measure the usage of their online resources.
COUNTER usage metrics
COUNTER (Counting Online Usage of NeTworked Electronic Resources): COUNTER provides the standard enabling vendors and publishers to supply their library customers with consistent, credible, and comparable usage data. Using COUNTER reports, libraries can get statistics about number of downloads, turnaways, and more. For more information about COUNTER reports, please review the COUNTER 4 to 5 Overview.
Libraries using a link resolver may be able to obtain and utilize link resolver usage data. This data can provide valuable insight into how their patrons are accessing and using library resources, including what resources patrons are using to access the link resolver.
Proxy server logs
A proxy server is a service that libraries use to authenticate their users to provide access to many online databases and publisher websites (Day, 2017). Libraries can utilize proxy server logs to learn more about how their users login to their systems, use the data to show how library resources contribute to student success, note excessive downloads, or see the most accessed journals. EXProxy, a service provided by OCLC, offers EZProxy Analytics software (for an additional fee) to libraries who use EZProxy. Additionally, Brian Erb, of the Florida Academic Library Services Cooperative, has developed a Libguide that explains how to use EZProxy login data for library assessment.
Google Analytics was originally created for commercial websites. However, libraries are now exploring strategic benchmarks, also known as key performance indicators (KPIs), which provide value to understand how users interact with resources from the library website. For example, KPIs can help libraries monitor user activities with research databases by looking at bounce rates, visits, selections per page view, average time on page, and visit depth (Yeager, 2017). More information can be found at the Google website.
How do we gather collection usage statistics?
There are several ways that libraries can access their usage statistics such as openly available resources, subscription tools that will aggregate usage data, or through their library management systems.
Third-party administrative sites
Most publishers (ProQuest, Springer Nature, IEEE, Elsevier, etc.) offer an administrative console for libraries to pull their own usage statistics. Along with publisher-developed sites, platforms with such reporting capabilities include:
Usage statistic aggregators (*requires a subscription)
There are several subscription and open source products that libraries can utilize to help them gather their usage statistics in one location:
- 360 Counter*
- Consortia Manager*
- EBSCO Usage Consolidation*
- EZProxy Analytics*
- Intota Assessment*
- Open Access Data & Analytics Tool (Delta Think)
- PlumX (for publishers)
- R5 Counter Harvester
- Springshare LibInsights*
- Unsub (formerly Unpaywall Journals)*
Library management systems
Many Library Management Systems (LMS) include access to analytics software that libraries can utilize. Examples of LMSs with this feature include, but are not limited to:
To see a full list and descriptions of LMSs, please review Marshall Breeding’s 2021 Library Systems Report.
Protocols and standards
Lastly, there are protocols and standards in place that guide and help define how library usage statistics should be generated, gathered, and reported. Two of the most well-known protocols and standards are from NISO (National Information Standards Organization):
- ANSI/NISO Z39.7-2013 Information Services and Use: Metrics & Statistics for Libraries and Information Providers Data Dictionary: This standard identifies categories for basic library statistical data reported at the national level, and provides associated definitions of terms, covering the areas such as reporting unit and target population, human resources, collection resources, infrastructure, finances, and services.
- SUSHI (Standardized Usage Statistics Harvesting Initiative): Standardized Usage Statistics Harvesting Initiative, also known as SUSHI, is a standard protocol (ANSI/NISO Z39.93-2003) that can be used by electronic resource management (ERM) systems (and other systems) to automate the transport of COUNTER formatted usage statistics. It can also be used to retrieve non-COUNTER reports that meet the specified requirements for retrieval by SUSHI.
References & resources
Arthur, M. (2018). Managing a comprehensive cost-per-use project in a large academic library. Serials Review, 44(4). https://doi.org/10.1080/00987913.2018.1558936
Association of College & Research Libraries. (2020). ACRL Academic Library Trends and Statistics: 2020 Survey Information. https://acrl.libguides.com/stats/surveyhelp
Baker, G., & Read, E. (2008). Vendor‐supplied usage data for electronic resources: a survey of academic libraries. Learned Publishing, 21, 48–57. Retrieved 15 September 2020, from https://onlinelibrary.wiley.com/doi/epdf/10.1087/095315108X247276.
Bernhardt, B. (2017). Leveraging vendor assessments of usage data. Serials Review, 43(3–4). https://doi.org/10.1080/00987913.2017.1369610
Blanchat, K. M. (2018). Beyond COUNTER-compliant: Ways to assess e-resources reporting tools. Serials Librarian, 74(1–4). https://doi.org/10.1080/0361526X.2018.1428464
Bergstrom, T., Uhrig, R., & Antelman, K. (2018). Looking under the COUNTER for overcounted downloads. UC Santa Barbara: Department of Economics. Retrieved from https://escholarship.org/uc/item/0vf2k2p0
Boukacem, C., & Schöpfel, J. (2012). Statistics usage by French academic libraries: A survey. Learned Publishing, 25, 271–278. Retrieved 15 September 2020, from https://onlinelibrary.wiley.com/doi/epdf/10.1087/20120406.
Bucknall, T., Bernhardt, B., & Johnson, A. (2014). Using cost per use to assess big deals. Serials Review, 40, 194–196. https://doi.org/10.1080/00987913.2014.949398
Conyers, A. (2010). Usage statistics and online behaviour. The E-Resources Management Handbook – UKSG. http://dx.doi.org/10.1629/9552448-0-3.2.2
Day, J. (2017). Proxy servers: Basics and resources. https://libtechlaunchpad.com/2017/04/25/proxy-servers-basics-and-resources/#:~:text=A%20proxy%20server%20is%20a%20service%20that%20libraries%20use%20to,online%20databases%20and%20publisher%20websites.
Duke University Libraries. (n.d.). Data visualization. https://library.duke.edu/data/data-visualization#:~:text=Data%20visualization%20is%20the%20term,our%20Duke%20Data%20Visualization%20LibGuide.
Glasser, S. (2018). Judging big deals—take two. Journal of Electronic Resources Librarianship, 30(1), 27–33. https://doi.org/10.1080/1941126X.2018.1443904.
Hamid, R. J., Nicholas, D., & Huntington, P. (2005). The use and users of scholarly e-journals: A review of log analysis studies. Aslib Proceedings: New Information Perspectives, 57(6), 554–571. https://doi.org/10.1108/00012530510634271.
Hinchcliffe, L. (2020). Taking a big bite out of the big deal. The Scholarly Kitchen. Retrieved 15 September 2020, from https://scholarlykitchen.sspnet.org/2020/05/19/taking-a-big-bite-out-of-the-big-deal/
Huffine, R. (2015). Going beyond cost-per-use: Assessing the value of purchased resources. Online Searcher, 39, 54–58.
Hulbert, L., Roach, D., & Julian, G. (2011). Integrating usage statistics into collection development decisions. The Serials Librarian, 60, 158–163. http://doi.org/10.1080/0361526X.2011.556027
Integrated Postsecondary Education Data System. (n.d.). Academic Libraries Information Center. https://nces.ed.gov/ipeds/report-your-data/resource-center-academic-libraries
Kaufman, P., & Watstein, S. B. (2008). Library value (return on investment, ROI) and the challenge of placing a value on public services. Reference Services Review, 36(3), 226–231. https://doi.org/10.1108/00907320810895314
Kelly, B., Hamasu, C., & Jones, B. (2012). Applying return on investment (ROI) in libraries. Journal of Library Administration, 52(8), 656–671. https://doi.org/10.1080/01930826.2012.747383
Kendrick, C. (2019). Cost per use overvalues journal subscriptions. The Scholarly Kitchen. Retrieved from https://scholarlykitchen.sspnet.org/2019/09/05/guest-post-cost-per-use-overvalues-journal-subscriptions
King, D., & Tenopir, C. (2013). Linking information seeking patterns with purpose, use, value, and return on investment of academic library journals. Evidence Based Library and Information Practice, 8, 153–162. https://doi.org/10.18438/B8B02M
Lamothe, A. (2014). The importance of identifying and accommodating e-resource usage data for the presence of outliers. Information Technology & Libraries, 33(2), 31–44. https://doi.org/10.6017/ital.v33i2.5341
Luther, J. (2008). University investment in the library: What’s the return? A case study at the University of Illinois at Urbana-Champaign. Library Connect White Paper. http://hdl.handle.net/2142/3587
Malapela, T., & De Jager, K. (2018). Theories of value and demonstrating their practical implementation in academic library services. The Journal of Academic Librarianship. 44(6), 775-780. https://doi.org/10.1016/j.acalib.2018.09.018
Mood, A. (2018). Statistics. Access Science. https://doi.org/10.1036/1097-8542.652400
Mezick, E. M. (2007). Return on investment: Libraries and student retention. The Journal of Academic Librarianship, 33(5), 561–566. https://doi.org/10.1016/j.acalib.2007.05.002
Nicholas, D., & Huntington. P. (2003). Micro-mining and segmented log file analysis: A method for enriching the data yield from Internet log files. Journal of Information Science, 29(5), 391–404. https://doi.org/10.1177%2F01655515030295005
Pan, D., Ferrer-Vinent, I. J., & Bruehl, M. (2014). Library value in the classroom: Assessing student learning outcomes from instruction and collections. The Journal of Academic Librarianship, 40(3–4), 332–338. https://doi.org/10.1016/j.acalib.2014.04.011
Pan, D., Wiersma, G., Williams, L., & Fong, Y. S. (2013) More than a number: Unexpected benefits of return on investment analysis. The Journal of Academic Librarianship, 39(6), 566–572. https://doi.org/10.1016/j.acalib.2013.05.002
Peters, T. (2002). What’s the use? The value of e-resource usage statistics. New Library World, 103, 39-47. https://doi.org/10.1108/03074800210415050.
Tay, A. (2017). 4 different ways of measuring library eresource usage. http://musingsaboutlibrarianship.blogspot.com/2017/02/4-different-ways-of-measuring-library.html.
Schufreider, B., & Romaine, S. (2008). Making sense of your usage statistics. The Serials Librarian, 54(3–4), 223–227. https://doi.org/10.1080/03615260801974164
Smith, K., & Arneson, J. (2017). Determining usage when vendors do not provide data. Serials Review, 43(1), 46–50. https://doi.org/10.1080/00987913.2017.1281788.
Smith, M. M., & Smith, J. A. (2016). What’s the use? A cost-per-use study of selected business databases. The International Information & Library Review, 48(1), 11–20. https://doi.org/10.1080/10572317.2016.1146037
Smulewitz, G., Celano, D., Andrade, J., & Lesher, M. (2013). ROI or bust: A glimpse into how librarians, publishers, and agents create value for survival. The Serials Librarian, 64, 216–223. https://doi.org/10.1080/0361526X.2013.761064
SPARC. (2020). Unsub gives libraries powerful evidence to walk away from big deals. https://sparcopen.org/news/2020/unsub-gives-libraries-powerful-evidence-to-walk-away-from-big-deals/
Tucker, J.C. (2009). Collection assessment of monograph purchases at the University of Nevada, Las Vegas Libraries. Collection Management, 34(3), 157–181. https://doi.org/10.1080/01462670902962959
Watson, K., Evans, J., Karvonen, A., & Whitley, T. (2016). Capturing the social value of buildings: The promise of Social Return on Investment (SROI). Building and Environment, 103, 289–301. https://doi.org/10.1016/j.buildenv.2016.04.007
Yeager, H. J. (2017). Using EZproxy and Google Analytics to evaluate electronic serials usage. Serials Review: North Carolina Serials Conference, 43(3–4), 208–215. https://doi.org/10.1080/00987913.2017.1350312