Designing LADs That Promote Sensemaking: A Participatory Tool

. Learning Analytics Dashboards (LADs) are visualisation tools built with the purpose of empowering teachers and learners to make purposeful decisions that impact the learning process. Due to their relatively recent emergence and the scarcity of studies on their design principles, learning dashboard design remains a major area of investigation in learning analytics research, and large scale diffusion to their stakeholders remains limited. We promote a participatory approach for designing LADs since their success in terms of acceptance and adoption is found to greatly depend on the degree to which its stakeholders have been involved in the design process. In this paper, we propose a comprehensive LAD design space, including a more precise definition of goals, social use, sensemaking, and visualization in order to guide the design process. We also present a tool to support the participatory design of LADs, which integrates our refined specification of the design space. First experiments in the context of a pilot study with teachers demonstrate that this participatory tool encourages group work, and in-depth exploration of LADs use.


Introduction
A Learning Analytics Dashboard (LAD) is a visualization tool built for the purpose of enabling teachers and learners to make relevant decisions that impact the learning process [17].Different indicators related to the learner(s), the learning process(es) and/or the learning context(s) are combined within a single display into one or more visualisations using different modalities, from textual and graphical representations to complex artefacts such as alerts and notifications that prompt action [27].Recent work underline that the term "dashboard" does not pick out a unique method of organizing, presenting, and using data, but rather covers a diverse set of practices [25].
Although LADs have received increasing interest in recent years, there is a general agreement that they are still under-researched and under-explored [2], large scale diffusion to their stakeholders remaining limited.We argue that reasons are multiple: (1) due to the their relative recent emergence, there is still a This preprint has not undergone peer review (when applicable) or any post-submission improvements or corrections.The Version of Record is published in Hilliger, I., Muñoz-Merino, P.J., De Laet, T., Ortega-Arranz, A., Farrell, T. (eds) Educating for a New Future: Making Sense of Technology-Enhanced Learning Adoption.EC-TEL 2022.Lecture Notes in Computer Science, vol 13450.Springer, Cham, and is available online at https://doi.org/10.1007/978-3-031-16290-9_54scarcity of studies on their design principles [11]; (2) designing effective LAD is a difficult process, in which stakeholders need to be involved.Indeed, LA tools may impose assumptions that do not meet the stakeholders' needs [8].(3) stakeholders are lacking experience of LAD use and data literacy.Many dashboard designs fail to measure the appropriateness of the incorporated visualizations for their visual literacy levels [27].Some authors emphasize the importance of empowering the stakeholders [7,17]; and (4) LADs often fail in turning insights into action since the processes by which people use these representations for insight seeking and decision-making are still not well understood [32].
Current research found that the success of dashboards in terms of acceptance and adoption, and more globally of any LA innovation, greatly depends on the degree to which its stakeholders have been involved in the design process [15].This has motivated the increasing focus of the LA research community on Human-Centred Design (HCD) approaches and the emergence of the Human-Centered Learning Analytics (HCLA) [5,9].Fig. 1: Interaction co-design process and roles for LA [22] Participatory design has recently emerged as a growing trend in education.Participatory design or co-design1 derived from user-centered design as a particular case of co-creation where designers who are trained in creativity work together with non-designers during the design process.Although LA academics and practitioners are increasingly acknowledging the relevance of HCD methods such as participatory design, their integration into learning analytics has been slow and is still not yet widespread [26], and approaches to achieving this remain unclear [10,3].
Figure 1 depicts a co-design process adapted to LA, proposed in [22], where activities are iterated, so as to refine the needs and get closer to the desired solution.Our aim is to instrument this process more specifically for LADs.In a previous work (citation omitted), Understand phase was established and has been continuously refined through extensive interaction with different stakeholders.In this paper, we focus more specifically on the Create phase.We propose a participatory design tool2 which aims to enable, promote and enhance the accurate and insightful expression of key design elements and requirements (including visualisation, idea generation, etc.).
Introduced by some authors (e.g., [29]) the notion of design space is a relevant support for ideation, creation, and evaluation.The design space also provides proper specifications for the subsequent convergent Deliver phase, proposed by [22] (see figure 1).As recent research on dashboards demonstrates, the sensemaking dimension is pivotal in the construction of relevant dashboards [21].We thus propose to include this dimension as part of the LAD design space.
The remainder of this paper begins with a review of relevant research.Section 3 details the design space we propose for LAD design.Section 4 introduces the tool we built for the participatory design of LADS.Section 5 briefly describes a case study that illustrates the use of the design tool in a participatory workshop involving teachers, before concluding.
2 Background and related work

Participatory design of LADs
The fundamental paradigm shift introduced by participatory approaches for design is a move from designing for users to designing with individuals as equal partners [24].In LA, it is defined an an approach where learners, educators, institutions, researchers, developers and designers are all included across different stages of the design process, from exploration to actual implementation [22].It promotes consensus building and the convergence of the different stakeholders on the main objective of the dashboard, encourages collective innovation and creativity, and anticipates possible adoption obstacles or usage difficulties.In LA research, while some examples of successful use for co-design of dashboards are reported in the literature [26], the LA community still lacks tools specific to the needs of LA stakeholders to effectively communicate and understand the design components [3,10].Popular methods being implemented include interview and focus groups, collaborative personas, and card-based co-design.
Workshops and focus groups are a common approach used to derive design ideas and identify stakeholder opinions.For example, Ahn et al. [1] used such direct co-design means in their partnership with college teachers for designing a LAD that meets the local needs of a particular educational setting.
Learning persona allows modelling and summarizing essential information about the people who are likely to be involved in the learning ecosystem.A learner persona is therefore a hypothetical learner who is representative of a certain number of potential learners, educators, etc. Personas must help answer three basic questions: what are the learner's needs, what kinds of limitations do they have, and what are their expectations?[34].In terms of co-design, people can be invited to build Persona profiles or create their own representations that can be merged later in different instances [22].
Card-based co-design is another popular co-design approach.Cards provide a common basis for understanding and communication between stakeholders and support creative combinations of information and ideas.Their use can enhance collaboration, combined creativity, and facilitate the exploration of possible connections between users' needs and the many proposed indicators and displays available in the design workshops [19].For instance, LA-DECK is a card-based co-design tool adapted to LA [3].It features suits of cards, grouped into three main themes: Context, Strategy, and Action.In another example of card-based approach, described in [19], two types of exploration cards are proposed: technology cards and domain cards.Technology cards represent technologies or interactive installations that may directly or indirectly be part of the design concepts.Typically, the types of visualization or interactions available in an LAD can be represented as technology maps.By contrast, domain cards represent information about the specific field for which the novel concepts are designed.Intended goals may be materialized thanks to domain cards.

Supporting sensemaking with LADs
LA Dashboards typically support and augment human cognition by offering visualizations of learning data [32].As it is important to know how the user makes sense of the information delivered with LADs, researchers start focusing on how sensemaking occurs with such tools [20].
Models are proposed to investigate interaction and sensemaking with LADs.They tend to break down the process into phases that go from perceiving the dashboard to taking and implementing actions.For instance, the model proposed in [31] defines four steps: awareness, reflection, sensemaking and action.The sensemaking framework of instructor analytics using dashboards elaborated in [33] starts by identifying the educational questions that lead to investigation and interpretation of the dashboard, followed by pedagogical responses.The process model for LA sensemaking through dashboards described in [6] evolves from one or more pedagogical events to its multiple data representations used by a set of sensemaking sub-processes (emotional, analytical, and intentional dimensions) to produce an interpretation of the events.The phases defined by these models are similar to the levels of situational awareness (SA) investigated by the naturalistic movement to explore human decisions [12]: perception of environmental elements in a volume of time and space, comprehension of their meaning, and projection of their state in the near future.Within this view that we adopt in this paper, sensemaking is the process of constructing situational awareness through which a course of action is developed [18].At the core of that research, interaction is investigated as the means by which a user interacts with the LAD to draw meaning.It is through interaction that a tight coupling is established between the analysis, the visualization and the human analyst.

LAD Design space
Following previous work (e.g.[23]), we seek to combine efficient tools that capture requirements and gradually collect them to build and enrich design models.To define a relevant support for the Ideation activity, bearing in mind that the expression of the need will be the basis for the production of prototypes, we need a design model, detailing main decisions relevant for stakeholders.Proposed by some authors (e.g., [29]) the notion of design space is a relevant support for ideation, creation, and evaluation.We adopt the following definition provided by Shaw [29]: "A design space identifies and organizes the decisions to be made, together with the alternatives for those decisions, thereby providing guidance for creating artifacts or a framework for comparing them." Some authors (e.g., [7]) propose to use 5 W's questions to address such a design space, focusing on the object of interest, from visualisation to dashboards and learning process analysis.In the context of LADs, we formulate the 5 W's as follows: (i) Who? depicts the audience and circulation between different stakeholders; (ii) When? permits to answer if usage is real-time or delayed; (iii) Why? corresponds to the goal of the LAD; (iv) What? details the context of LAD's usage, and relevant data; and (v) HoW? focuses on visualization, which is related to sensemaking with a LAD.Table 1 summarizes the properties we identified of each of these design dimensions.The next paragraphs detail and discuss each of these dimensions.[28,16].Acknowledging that dashboards may be a communication tool between different stakeholders, the target audience concept broadens the notion of user [25].
Circulation.Related to audience, authors detail interpersonal circulation of a dashboard based on four groups: Public, Organizational, Social, Individual, each becoming more specific and requiring more context (where the necessary context might not be included in the dashboard itself) [25].A public dashboard is intended for general consumption, and may describe societally-relevant data.Dashboards for organizations are broadly relevant within an organizational construct for many different individuals who share a common goal (say, validating diplomas).Social circulation captures contexts in which an individual restricts the access to the dashboard to individuals of their choosing, identifying scenarios of sensitive data or analysis.Individual circulation captures dashboards that quantify the individual and are generally not shared, except with trusted individuals (e.g., a teacher for a learner).
When?This aspect of the design space depicts the time of use of the dashboard.Associated with audience is the idea that users and data come together.It is possible to qualify whether the communication is based on real-time data processing, i.e. whether the communication is based on what is currently happening, or is based on historical data.
Why? Refining LAD Goals.This dimension is broadly related to the intended purpose of the dashboard.Although this is a central aspect of LADs that provides the rationale for the design, it remains unclear when referring to the existing literature.Some authors refer to purpose, others to goals, or even objectives.We retain the term goal.We argue that the goals of LAD refer to two dimensions of the design space: focus and situation awareness level.These two dimensions make it possible to integrate all the existing options, and make it possible to consider new ones.We argue that (i) it opens new opportunities for design, and (ii) it encourages designers to go beyond simple options like "monitoring", and to explore higher levels of situation awareness.
Focus.Most works on dashboard goals provide one or more lists of examples [17,28].In [28], the authors analysed students objectives under the perspective of aimed intervention (or feedback), and proposed the following focuses on learning process: (i) cognitive, (ii) outcome oriented (e.g.achievement level), (iii) processoriented, (iv) behavioral, (v) meta-cognitive.As social presence is a key element of educational experience [14], we propose to add to this list (vi) the social aspect, that relates to group-work or learner relations (for example in forums).Interventions of (vii) emotions [13] should complete this list.Teachers goals include interventions concerning the students' learning process, and the same focus list as students apply.They are also concerned by course management, and their goals may focus on (i) people (e.g.students at risk), (ii) resources (management or improvement) , (iii) activities (including assessment), (iv) experience.Other stakeholders at mega-or macro-level may be interested by additional focus objects, but this list remains to be proposed.
Situational awareness level.Typically a goal is expressed as a verb followed by an object.When browsing goal lists, the expression may be related to a situation awareness level or state [12] : (1) Monitor or awareness, (2) Analysis, or reflection, (3) Projection, or prediction, (4) Action, or decision, intervention, impact, feedback, assessment.
What?The Dashboard's data.According to the goal defined and the intended audience, a collection of data will be necessary to support the sensemaking (i.e.comprehension) process of targeted users.Data scope must be defined, as it is decoupled from the audience.It may concern a learner, the classroom (or a whole cohort) or even the institution, but it may also concern the teacher.Including the teacher in this dimension encourages to investigate teacher analytics [4] where the goal is to improve the learning process itself.
HoW? Visualization and Sensemaking in LADs.HoW is a central question for dashboards, as relevant visualizations are required to gain insights [25,7].As identified in the "Why" section, dashboard may support a sensemaking process, where the user(s) may navigate across different levels of sensemaking and may need to analyse some data.Multiple panels, interactions, and additional features, such as alerting and notifying are relevant to qualify the dashboards.Regarding interactions, a simple but supportive taxonomy is proposed in [30].Known as the information-seeking mantra, this taxonomy summarizes the essential elements of interacting with graphically presented information by defining a high-level set of tasks: (1) Overview task purpose is to provide the user with a global view of the available data and to display aggregated, summarised and less targeted representations of this data; (2) Zoom task is executed to investigate a part of the data by allowing to select this part and to interact to select the focus and the zoom factor; (3) Filter task reduces the amount of data and/or visual objects displayed, and helps the user find and focus on specific items of interest; (4) Details-on-Demand task refers to the use of techniques to obtain more precision on the data in order to obtain a better insight; (5) Relate task allows users to view relationships between data points; (6) History task allows keeping a record of actions to support undo, replay, and progressive refinement; and (7) Extract task allows users to visualize a part or sub-collection in order to focus only on the data that is necessary for immediate use.

Description of the participatory design toolkit
In the previous section, we detailed LAD design space, as a framework to organize decisions to be made.To be effective, a participatory design method needs to be properly instrumented.We therefore need to identify tools that can support dashboard users in better expressing their expectations and needs.In this section, we propose to combine suitable tools according to the dimensions of the design space to foster meaningful participatory creativity.Our aim is to provide support for collaborative workshops.We distinguish three phases to support the process: (1) Identification of the LAD's context and goal; (2) Data & Visualisation to explore useful data; and (3) Sensemaking Sketchup to explore LAD organisation and interaction to support sensemaking.These phases are materialized by dedicated boards that group various cards representing the dimensions of the design space (figure 2).Identification board.This board gathers the Who?, When? and Why? questions.A persona form enables to personify the stakeholders and to collect relevant information, such as expertise, visual literacy, or motivation.The goal being pivotal to guide the ideation process, we dedicate a specific domain card to support goal expression.Participants are invited to express their own goal and to relate it to focus and situation awareness level.Those lists may be adapted to participant profiles, being expressed differently whether they are learners, teachers, or other types.Additional domain cards are proposed to inform additional information: Circulation, answer to When? question, Data Scope.
Data & visualization board.This board is intended to answer to the What?question.Participants are invited to identify relevant data that are useful to attain their goal.They fill a specific data card for each.They are also invited to associate visualisation they feel relevant.For this, a set of technology cards proposing classical visualizations is provided.
Sensemaking Sketchup board.As LAD is a visual object, participants are willing to sketch a representation of aimed LAD.Proposing a sketching phase in a participatory workshop is essential.Sensemaking is supported in three ways.First, in order to foster browsing of the different levels of Situation Awareness, mockups of different colours have been provided: red for monitoring, blue for analysing and green for action.Participants are invited to associate data and visualisations to the different mockups.Second, technology cards of the main interaction options are provided to help participants express how they wish to interact with the LAD.Third, a storyboard form is attached to each mockup inviting them to express how they imagine the sensemaking process.

Ideation workshop using the toolkit
The design session starts with negotiating a goal and setting up the persona (figure 3a).Next, they should work collaboratively to make use of the various boards, cards, and other layouts designed to facilitate the expression of their needs, and to support their creativity.The sequence in which these actions are addressed does not matter, as the participants may have prior ideas (data you want to use, a dashboard you want to use...).Nevertheless, the following order is of interest by default: Who wants to do what, with what data and how to access it to track the achievement of a goal, understand what is happening and act to better fulfill their goal.The more content users can express, the more readily the corresponding dashboard can be created.If they are not inspired by a particular section at a given time, they should not get stuck on it; they can come back to it later.Finally, the session resulted in a potential design represented as filled-in cards arranged in the different boards (figure 3b).
5 The proposed toolkit in practice: a participatory design workshop Ideation session.The main challenge encountered during the ideation session was the negotiation process necessary to establish a persona.This reflects the different and sometimes conflicting personality traits, challenges, needs and aspirations of the participants.One participant stated: "Certainly, the persona is a central concept when designing, as it sets the direction.Nevertheless, it is difficult to know from the start what level of detail in the description would be necessary".This indicates some misunderstanding of the purpose of the persona, and highlights a need to better qualify the elements of the persona.Once the description of the persona has been established, the participants moved on to the definition of the pursued goal.They agreed to consider learning progress, to focus on the process with a situational awareness level going from monitoring to planing.Their aim was to adapt their teaching according to the obtained feedback and to develop equality among students.Nevertheless, as stated by a teacher, "a dashboard can have a goal, but sometimes this latter is much more related to a use than to the dashboard: one can divert for another use a dashboard preconceived for a specific goal." In accordance to the specified goal, the participants used the context description card to express the willingness to consider in-class data of students of each session individually and in combination, and to share the dashboard with all the teaching staff and to use it.They also noted the impossibility for them to use the dashboard in real-time."these cards are straightforward as they allow us to clearly establish the perimeters, stated one participant.The clear specification of the identification board helped the group in building the target picture using the DataViz board, and simplified the choice of data and visualisation to be used.Nevertheless, the different levels of visual literacy have led to debates about which visual representations are most appropriate.The participants felt and expressed the need to be supported in this phase, as stated by the majority of the participants: "it is important to involve an expert graphical designer in the co-design process" Finally, for the construction of the different views of the dashboard following the reasoning stage, once the participants had understood the rationale, it made the design much easier.Indeed, they found this approach to conceptualizing a dashboard intuitive since it reflects and even materializes the steps of the reasoning and allows them to project themselves into real use scenarios."The possibility to project the reasoning process into visual representation is appealing, though it seems to add complexity having to imagine the different screens.Sometimes, a single screen is enough for all the steps".Demonstration session.Based on the insights expressed during the ideation phase, we (researchers) moved on to create prototype dashboards that implement what the teachers described (figure 3c).Drawing on the discussions we observed during the first session, we included in our prototype proposals other ways (data and visualisations) to improve the achievement of the defined objectives.The results were presented two weeks after the first session.The presentation was accompanied by a discussion about the use of the dashboard, the impact of this use and more generally how to assist teachers to be aware of their lessons and students' needs and to make informed decisions.Participants found the whole strategy of participatory design as very fruitful as it leads to the emergence of a passionate debate and the expression of original ideas about the different strategies that could be followed to improve learning in the classroom.

Conclusion
In this contribution, we have enhanced the LAD design space, by promoting a more precise decomposition of the intended goals, including situation awareness level.Based on design space dimensions, we propose a novel participatory tool specifically designed to support LAD co-design3 .We combine personas profile to express user needs, ideation card to promote domain needs, and sketching to enable prototyping.The proposed design space has also been streamlined in order to facilitate generative development of functional prototypes, enabling further exchanges more focused on design issues and usability.This step is essential to ensure that ideation may conduct to real use, and adoption of learning analytics.Seamless integration of participatory design and generative tools in LAD design still requires further investigation.
A first experiment has been conducted as proof of concept.It demonstrates that innovative proposals and LA adoption are possible with teachers, using a participatory approach.The research team involved in this effort aims to specialize and test the tool in different contexts, with different audiences, and for different purposes.We strongly believe that this kind of tools are contextual.For example, level of situation awareness may be expressed as monitoring, analysis, and decision-making at a governance or institutional level, but will rather be: awareness, reflection and feedback for the student.Adoption of different participatory tools may also vary according to different audiences.
To conclude, collecting LAD proposals from users and practitioners may bring out new needs and may conduct to identify new intended goals that should be shared with the learning community.
activity, its context, what is being observed, over what period of time, and the required data Measure Data Indicate the activity, its context, what is being observed, over what period of time, and the required data Measure Data Indicate the activity, its context, what is being observed, over what period of time, and the required data Measure Data Indicate the activity, its context, what is being observed, over what period of time, and the required data Mea sure

Fig. 3 :
Fig. 3: A participatory design workshop using the proposed toolkit

Table 1 :
Dimensions of the LAD design spaceWho?Identify Audience.Two dimensions are related to the audience: (1) LAD stakeholders identification, which is classical in LA; and (2) dashboard circulation, which acknowledges dashboards as communication tools.