NORMalize: A Tutorial on the Normative Design and Evaluation of Information Access Systems

Johannes Kruse, Lien Michiels, Alain Starke, Nava Tintarev, Sanne Vrijenhoek

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review


Information access systems, such as Google News or YouTube, increasingly employ algorithms to rank diverse content such as music, recipes, and news articles. Acknowledging the influential role of these algorithms as gatekeepers to online content, the research community is increasingly exploring ‘beyond-accuracy’ metrics. However, deciding what norms and values are relevant and should be prioritized when designing and evaluating information access systems is a challenging task. This tutorial aims to cultivate normative thinking and decision-making in the design and evaluation of information access systems. The tutorial comprises two key components. The first part involves a lecture on the foundational principles of normative thinking, emphasizing the importance of reflecting on the desired state of a system rather than its current state. The second part is an interactive session where participants engage in group discussions, applying normative thinking to a specific use case. Participants analyze the system’s usage, stakeholders, and relevant norms and values and address potential conflicts between stakeholders and/or values. Through a point-allocation exercise, participants represent stakeholders and advocate for specific values, fostering a deeper understanding of normative decision-making in the context of information access systems.
Original languageEnglish
Title of host publicationProceedings of the 2024 Conference on Human Information Interaction and Retrieval (CHIIR ’24)
PublisherAssociation for Computing Machinery
Publication date2024
ISBN (Electronic)979-8-4007-0434-5
Publication statusPublished - 2024
Event2024 Conference on Human Information Interaction and Retrieval - Sheffield, United Kingdom
Duration: 10 Mar 202414 Mar 2024


Conference2024 Conference on Human Information Interaction and Retrieval
Country/TerritoryUnited Kingdom


  • Normative thinking
  • Normative design
  • Norms
  • Information systems
  • INformation access systems
  • Recommender systems
  • Information retrieval


Dive into the research topics of 'NORMalize: A Tutorial on the Normative Design and Evaluation of Information Access Systems'. Together they form a unique fingerprint.

Cite this