Understanding and Evaluating Search Experience

This book is intended for anyone interested in learning more about how search works and how it is evaluated. We all use searchit's a familiar utility. Yet, few of us stop and think about how search works, what makes search results good, and who, if anyone, decides what good looks like. Search has a long and glorious history, yet it continues to evolve, and with it, the measurement and our understanding of the kinds of experiences search can deliver continues to evolve, as well. We will discuss the basics of how search engines work, how humans use search engines, and how measurement works. Equipped with these general topics, we will then dive into the established ways of measuring search user experience, and their pros and cons. We will talk about collecting labels from human judges, analyzing usage logs, surveying end users, and even touch upon automated evaluation methods. After introducing different ways of collecting metrics, we will cover experimentation as it applies to search evaluation. The book will cover evaluating different aspects of searchfrom search user interface (UI), to results presentation, to the quality of search algorithms. In covering these topics, we will touch upon many issues in evaluation that became sources of controversyfrom user privacy, to ethical considerations, to transparency, to potential for bias. We will conclude by contrasting measuring with understanding, and pondering the future of search evaluation.

Autorentext

Maria Stone spent years in different roles spanning user research and evaluative data science. Her search career spans two decades in Silicon Valley, where she worked for Alta Vista, Google, Yahoo, and Microsoft, as well as Apple Maps and Siri. Prior to her work in industry, she was an academic researcher and lecturer focused on studying human memory and attention, where she holds a Ph.D. in Cognitive Psychology from UC Berkeley and has authored publications in Cognitive Psychology, HCI, and Information Retrieval. She currently works for Spotify and resides in Stockholm.

Inhalt
Acknowledgments.- Introduction.- How Search Engines Work.- How Searchers Think About Search.- How Measurement Works.- Components of Search that Can Be Evaluated.- Units of Analysis: Query, Task, User.- Types of Metrics for Search Evaluation.- What Gets Left Out: Understanding.- References.- Author's Biography.