PRE-CONFERENCE WORKSHOPS

Workshop Facilitators: A/Prof Margaret Bearman, Prof Elizabeth Molloy

Feedback conversations are important for learning through assessment but unfortunately they often don’t deliver on this potential. Despite literature espousing active learner involvement in feedback, ‘interactions’ often take the form of well-intended supervisor monologues. This workshop describes the phenomenon of intellectual candour or “…disclosure for the purpose of one’s own learning and the learning of others” (Molloy and Bearman 2018, p 1). We outline how educators judiciously disclosing their own uncertainties may influence feedback conversations for the better. In this workshop, illustrative examples of workplace assessment conversations will be presented, foregrounding how the healthcare assessment milieu influences learners reluctance to render themselves vulnerable. Participants will experiment with how ‘the moves’ of the educator can promote or stifle learner engagement in feedback, with a particular focus on intellectual candour.

Workshop Facilitators: Dr Carlos Gomez-Garibello, Dr Maryam Wagner and Dr Paola Fata

With the transition to competency-based medical education (CBME), training programs are incorporating Entrustable Professional Activities (EPAs) in their curriculum, teaching and assessment. Whilst the use of EPAs is becoming increasingly ubiquitous, educators do not always capitalize on the opportunities to integrate EPAs throughout their programs. Even less frequent is the use of EPAs to develop test blueprints to test clinical knowledge. We propose using an Evidence-Centred Design (ECD) framework to use EPAs to design assessments and/or tests in a systematic and rigorous manner. The workshop will be delivered through plenaries, hands-on activities, and discussions. Part 1 will be devoted to teaching the basics of the creation of test blueprints, and the educational tenets of EPAs and their components. Part 2 involves illustrating an evidence-based approach (Mislevy et al, 2003) to use EPAs to develop a test blueprint. The process will be exemplified using the Canadian Association of General Surgeons’ (CAGS) formative exam. During Part 3 of the workshop, participants will be provided a selection of EPAs, competencies, and a test blueprint template, so they can apply the principles covered in Part 2 to develop test blueprint. Part 4 will engage participants in a discussion of how to apply the ideas and principles in their educational contexts.

Workshop Facilitators: Dr Nidhi Garg, Tyler Clark, Dr Lauren O’Mullane and Prof Deborah O’Mara

Programmatic Assessment (PA) is increasingly being implemented in medical and health professional programs throughout the world since first being outlined by Schuwirth and Van der Vleuten (2011). While there is widespread support for the principles of PA, studies to-date have been largely theoretical and some medical schools have implemented only some aspects of PA (Torre et al., 2020). Unintended consequences of PA can include an increase in data complexity, staff workload, resource requirements, persistent failure to fail, non-transparent decision-making and increase in student stress (Ryan et al., 2023). Sydney Medical School implemented PA in 2020 alongside a curriculum reform. While there have been benefits to implementing a PA system, challenges also arose. This necessitated adaptations to the initial vision of PA to suit our context in which allow for progression decisions to be made for multiple large cohorts at a single time point.

Workshop Facilitators: Dr Gil Myers, Dr Alison Sturrock, Prof Chris McManus

In-depth information on how candidates respond to single best answer items during a real time-limited examination has been made possible with the advent of computer-based tests(CBT)(1). We have developed a programme to create a “ClickStream” (2) of keyboard/mouse movements. This allows us to visualise how students move between questions, view images, select answers, revise answers, and return to questions. Due to the precise timing of events and volume of data produced, it is challenging to interpret these events so we have created visual “ClickMaps”(3) of behavioural patterns, allowing us to quickly evaluate detailed maps of candidate behaviour. We will discuss our ClickMap and inferences, and outline different strategies used by candidates. We have evidence from multiple student groups about how candidates behave and the effect that their methods have on their outcomes. As collusion is a worry in high-stakes assessments, the introduction of CBT has made this worry more acute. We will examine our “ClickMaps” patterns in light of Acinonyx data and discussions around cheating.

Workshop Facilitators: Dr Shelley Ross, Dr Brent Kvern, Dr Alison Baker, Dr Cheri Bethune, Dr Kathrine Lawrence, Dr Keith Wilson, Dr Erich Hanel, Dr Annelise Miller, Dr Theresa van der Goes and Dr Karen Schultz

Many health professions education programs globally are in the planning or implementation phase of adopting competency-based medical education (CBME). Concurrently, training programs are considering ways to improve approaches to assessment to better capture evidence about the competence of their learners. Programmatic assessment, first introduced to health professions education by van der Vleuten and Schuwirth in 2005, is an approach to assessment that shifts the focus away from summative examinations. Instead, the focus is on the longitudinal collection of multiple pieces of data about learner competence, collected through multiple different assessment tools. While many programs have embraced the idea of programmatic assessment, designing effective and trustworthy programmatic assessment is a serious challenge for many programs. How can those who plan assessment for health professions training programs ensure that they are assessing the right things, with the right tools, in the right way? Many programs have already done the work of identifying and describing the professional competencies that must be assessed during training. The next steps are to determine 1) which assessment tools are most appropriate to assess each competency, and 2) how to make sense of the assessment data that is collected. This interactive workshop will give participants the opportunity to work through the process of developing programmatic assessment, guided by a template in the form of a worksheet. Using the case example of Canadian family medicine training, participants will identify the enablers and barriers faced by accrediting bodies and individual programs in transforming assessment. We will introduce the basic principles of programmatic assessment, and give scenario prompts to facilitate frequent small group breakout discussions to consolidate the information shared and make it actionable. Using the worksheet, individuals and small groups will look at how to operationalize the didactic information about what factors to consider in designing programmatic assessment.

Workshop Facilitators: Dr Laura Culver Edgar, Raghdah Al Bualy and Dr Eric Holmboe

Assessment remains a major challenge for programs and faculty, especially for the competencies of professionalism, interpersonal and communication skills (e.g., teamwork), systems-based practice (e.g., quality improvement and patient safety, roles within the healthcare system), and practice-based learning and improvement (e.g., evidence-based practice/scholarship/life-long learning). All programs accredited by the Accreditation Council for Graduate Medical Education (ACGME) and Accreditation Council for Graduate Medical Education – International (ACGME – I) are required to assess a similar set of subcompetencies, referred to as Harmonized Milestones, under each of these 4 core competencies. Using evidence-based principles of “good assessment” and methods in conjunction with faculty development in these essential competencies can improve overall programmatic assessment. The session will contain three sections. The introduction will enable participants to learn about the evidence-based process used to create the Harmonized Milestones and how these subcompetencies map to the medical education frameworks of other systems and countries. The second section will provide an overview of the principles of “good assessment” and utility for assessment methods used for each. Participants will discuss these tools and share how they have assessed these key competencies in their context. The final section will review how to implement these tools through the creation of shared mental models. The session will close with small group discussion with report out and discussion on how programs can implement these tools and meet the challenges of assessing each of these competencies.

Workshop Facilitators: Dr Ruth Hew, Dr David Mai and A/Prof Victor Lee

The Case based Discussion (CbD) or chart stimulated recall, is a well described work-place based assessment (WBA) tool. In its most basic form it can be used to identify concrete knowledge gaps. Used optimally, CbDs can help tease out a learner’s clinical reasoning and decision-making, identify biases and guide further learning. This makes the CbD an important tool in the assessment toolkit. The logistics of the CbD are as follows: 2-4 cases are identified by the learner and forwarded to the assessor who then chooses one for discussion. The trainee presents a summary and synthesis of the case, the assessor probes for their clinical reasoning and the trainee can demonstrate their reflective skills, clinical decision-making and identify any potential biases. As the original documentation forms part of the assessment, the assessor can also comment on the trainee’s documentation skills. In this interactive workshop, participants will be able to explore the components that go to make up an effective learning encounter using a CbD. The workshop conveners will use facilitated small group discussions, role plays and video simulated rating and calibration exercises to engage participants in the exploration of the important components of the CbD, to illustrate how to optimise the utility of the tool and demonstrate how to use it as a tool for both learning and assessment.

Workshop Facilitators: A/Prof Diantha Soemantri and Rita Mustika

Professionalism is at the core of health professionals’ competencies; however, significant challenges still exist, especially in terms of how best to assess it. Since assessment drives learning, discussion, and studies on how to assess professionalism are still ongoing. Continuous feedback is currently considered as the heart of assessment, aligned with the principle of programmatic assessment. Therefore, the assessment of medical professionalism should also take into account the importance of feedback. Formative assessment is one of the means to provide actionable feedback that students can use to improve their learning, in this case, their professionalism attributes. Sargeant et al (2015) developed a feedback model called R2C2, which consists of four phases, build relationships, explore reactions, explore content, and coach for performance change. This model facilitates students to reflect on the feedback. In the last phase of the R2C2 model, students will be coached using triggering questions to come up with action plans for continuous improvement. The coaching phase is then essential; hence we propose to combine the R2C2 model with GROW coaching model, which stands for Goal, Reality, Options, and Will. We believe that the GROW model will augment the coaching phase of R2C2, given more thorough, in-depth, and structured ways to coach students to set goals, explore current situations and options for moving forward, and agree on specific action plans. Therefore, the R2C2-GROW model will enhance the feedback delivered within the formative assessment of professionalism.

Workshop Facilitators: Dr Elizabeth Kachur, Dr Alice Fornari and Dr Thanakorn (TJ) Jirasevijinda

Global efforts to promote Diversity, Equity and Inclusion (DEI) have highlighted the need for bias-reduction education and assessment at all training levels. This includes clinicians in practice who need these skills in patient care as well as teaching. Biases can target any personal characteristics, from race and gender to sexual orientation and immigration status. Whether they are implicit or explicit, they will affect our interactions with patients, families, learners, and colleagues. Medical professionals and teachers bear the responsibility to maintain awareness and work to mitigate the adverse effects of biases. Formative Objective Structured Clinical Exams/Exercises (OSCEs) have proven to be effective and efficient training tools for addressing complex interpersonal situations. The same is true for Objective Structured Teaching Exercises (OSTEs) that have become a quintessential faculty development tool. Whether they are for trainees or faculty, station encounters can lead to multi-source feedback and debriefings to support best practice strategies. Over the years a variety of OSCE and OSTE stations have been developed to address racism and other biases. They can be categorized as focusing on 1) patient encounters (e.g., sequelae of historic racism); 2) encounters with learners and colleagues (e.g., allyship related to gender discrimination); 3) managing bias and microaggressions when learners themselves are the target (e.g., rejection of care providers because of their personal characteristics). These are difficult stations for everyone involved (including simulated/standardized participants – SPs), but with adequate pre- and de-briefing, they can be a powerful preparation for future real-life scenarios. This Pre-Conference Workshop will address the opportunities and challenges of each station type. The goal is to help participants create and implement bias-related OSCE or OSTE stations that fit their particular program.

Workshop Facilitators: Dr Beth Bierer, Prof Elizabeth Molloy and Brownell Anderson

The role of bias in assessment has long been recognized as a source of measurement error. Recently, the emphasis on bias in assessment has focused on gender and race/ethnicity. The association between being an introvert (or being quiet as a sign of cultural respect) on performance assessment ratings has received minimal attention in the medical education literature, particularly in graduate medical education (GME). The American Psychological Association defines introversion as a personality trait which, like extraversion, exists as a continuum of attitudes and behaviors. Introverts often appear reserved and are socially and cognitively more reflective than their think-aloud extraverted colleagues. Available studies reveal that introverts receive lower scores on interpersonal behaviors in clerkships. This session will incorporate a difficult teaching/learner case discussion to delineate strategies to share with faculty and introverted learners faced with this assessment challenge. The session will utilize a modified “Morbidity and Mortality” format focused on an actual case of a third year resident who, during their residency exit interview, revealed the perception that their naturally shy/quiet demeanor led to inaccurate assessments of their medical knowledge and decision-making ability. The participants will discuss the case and will be given the opportunity to ask for additional information before generating hypotheses, and articulating what they would say/do during the resident’s exit interview as well as possible long-term solutions for others. There will be a brief summary of the available literature from business and medical education regarding introverts’ behaviors and associated performance assessments.

Workshop Facilitators: Dr Kimberly Dahlman, Dr Alana Newell, Prof Neil Osheroff and Dr Nancy Moreno

Logic models are valuable tools for program planning and evaluation. A logic model visually connects intended outcomes with the resources and activities that go into a program, depicting relationships among program inputs, planned work, measurable outputs or products, and short- and long-term outcomes or impacts for students or other learners. Thus, a well-constructed logic model can guide decision-making about data collection, assessment timepoints and overall program evaluation, supporting clear alignment between these measures and the desired outcomes. Importantly, these assessments and measurements need to be inclusive and multisource, considering near- and distal-peer and self-assessments. In this IAMSE-organized session, an experienced investigator and program evaluator team will guide participants through each of the elements of a logic model, and provide templates, strategies, and practice in the development of well-aligned education program logic models.

Workshop Facilitators: Dr Maryam Wagner and Dr Carlos Gomez-Garibello

Recent and sustained developments in generative artificial intelligence (AI) tools, and their widespread accessibility are challenging typical methods for assessment of learners in health professions contexts. Learners are using AI tools such as ChatGPT to brainstorm ideas, answer multiple choice questions, and submit both edited and non-edited written materials (Hao, 2023). The pervasiveness of these tools and their uses have prompted many discussions about their impact on assessment- both positive and inciting challenges. The workshop will be delivered through mini-plenaries, hands-on activities, and discussions. Part I of the workshop is a short plenary presentation to introduce the audience to various capabilities of artificial intelligence tools that learner assessment. Part II of the workshop will engage participants in a jigsaw activity that guides participants to explore key issues and topics of assessment in the context of AI, its risks and challenges, and ethical and pragmatic implications for workplace-based assessment and research. The workshop will conclude with Part III with a facilitated discussion to synthesize the outcomes of the workshop.

Workshop Facilitators: Prof Brian Jolly

In 2012 Jim Crossley and I published a paper(1) with this title. It was a work that tried to condense the research on and ideas about WBA at that time around 4 main concepts. First that the right questions be asked; second that they needed to be phrased appropriately, and third, they should be about important and measurable features, and finally they should be directed towards the right people, i.e. those that would know about those measurable features. Depending on which database is used it has been cited between 250 and 300 times. This interactive workshop will unpack those ideas along with a narrative that describes how the paper nearly did not get written, and when it did some of its ideas became uncomfortable for some clinicians and academics. It raised the issues of interprofessional contribution to assessment, and how the questions should be expressed in the best way (not always agreed upon) to evaluate the particular skills or domain of interest. This workshop will critically reflect on the premises in the original paper in light of the emerging evidence and feedback from the application of WBA over the last decade. For example, we did not consider the patient’s voice in the assessment process. Participants will be given opportunity to outline their challenges in assessment and whether adopting WBA is feasible, and any successes they cherish. The four concepts will be outlined, and their usefulness discussed. The workshop will aim to help participants to refine their own interests and pursuits (for example as researchers, to shape the research questions that still need to be addressed about WBA, or as practitioners on how they might develop their practice).

Workshop Facilitators: Prof Russell D’Souza, Prof Mary Mathew, Prof Princy Palatty and Prof Joseph Thornton

Medical ethics involves the application of moral rules to situations exclusive to the medical world. It warrants sound moral reasoning that originates from the fundamentals of bioethics. Bioethics encompasses a multitude of disciplines that, includes medical health and law. Its execution involves personnel from various specialties like physicians, researchers, lawmakers, politicians, and social scientists. It is not just sufficient to have mere knowledge and technical skills but to have a profound understanding of bioethics principles, professionalism, and communication skills. The students were expected to “catch” the bioethical aspects that were hidden within the curriculum. The current assessment strategies aim to abide by Bloom’s taxonomy and test the awareness and knowledge gained by the student. Imparting bioethics in medical and health professions education is an art, and it is challenging to figure out the most appropriate method of instruction. Any educational system relies on assessment. There is no single effective assessment tool for bioethics and professionalism. Using diverse tools and assorted raters in multiple settings would give a clear picture of what the student has captured. The assessment of competencies in bioethics relates to the Cognitive, Psychomotor (behavioural) and Affective domains. The assessment of the cognitive and psychomotor domains is amenable to assessment, with tools available. Still, the affective domain poses a greater challenge in validating bioethics as it relates to internalization and actual practice, which is difficult to assess. A multi-modal approach is necessary to assess bioethics in medical and health profession education from varied situations. This workshop will offer newly developed formative and summative assessment strategies and tools to evaluate bioethics competencies in medical and health profession students in the affective, cognitive, and behavioural domains. The tools for the assessment of bioethics presently involve Standard assessment tools. This workshop will introduce the developed and validated Innovative assessment tools. This interactive workshop will have three parts. Part one will focus on the assessment tools. Part two will deal with the assessment implementation. Part three will concentrate on the analysis of the assessment. Participants will be given all the tools for assessment, and the groups are at liberty to choose an assessment strategy and administer it. They will debrief together on the analysis of the tool and then be assessed.

Workshop Facilitators: Prof Anna Ryan, Dr Mike Tweed, A/Prof Glendon Tait and A/Prof Suzanne Schut

A programmatic system of assessment is focused on longitudinal delivery of authentic assessment events involving different assessment formats where data is accumulated against a meaningful framework. Designed to address some of the problems associated with traditional assessment systems, programmatic assessment aims to increase student engagement, reduce failure to fail, and provide rich feedback to support learner growth and development, while also allowing robust progress decisions. In most contexts where traditional assessment approaches are in place, a shift to programmatic assessment involves significant change. While the outcomes of such change should have expected and intended consequences, such change can also lead to unintended and undesired consequences such as increased assessment workload, increased student anxiety, unlimited opportunities to meet standards, devaluing of individual assessments (or topic areas) and unwieldly complexity of feedback and decision-making data.