Technological educational products for teacher training in educational assessment: report of experiences in Medical graduationABSTRACT
Introduction: Medical schools expansion poses challenges to teacher training, including the systematization of processes such as assessment, which directly impacts the quality of medical training. Aligned with the competency levels of Miller’s Pyramid, assessment ensures both the acquisition of theoretical knowledge and the development of practical and attitudinal skills. The incorporation of andragogical knowledge into this process is still a gap. In order to fill it and also to enhance teacher training in undergraduate medical courses, technological educational products have been developed to guide the preparation of test items in a formative educational assessment process.
Objective: Report an interdisciplinary collaborative experience, which involved medical educators (domain experts), linguists (communication experts), librarians (information science experts) and an audiovisual team (technical experts).
Methodology: Under andragogy principles, adults learn better when they have the autonomy to follow personalized paths. Thus, a learning path was created, which describes processes, activities and agents. Guides and scripts were created, an evaluation committee was set up and webinars, workshops, a digital platform for tests, video animations and an online course were produced.
Results: Educational products were incorporated into the teacher training process to guide the preparation of test items.
Discussion: Although these products may favor the systematization of assessment, resistance to their adoption and institutional limitations for their implementation indicate that simply making technologies available does not guarantee effective changes.
Conclusion: Still, results point to progress in raising teachers’ awareness of a more careful formative assessment for building an integrated assessment culture.
Keywords:
medical education; competency-based teaching; formative assessment; tests; interdisciplinarity
RESUMO
Introdução: A expansão das faculdades de Medicina impõe desafios à formação docente, entre os quais está a sistematização de processos como o avaliativo, que impacta diretamente a qualidade da formação médica. Alinhado aos níveis de competência da Pirâmide de Miller, o processo avaliativo garante não só a aquisição do conhecimento teórico, mas também o desenvolvimento de habilidades práticas e atitudinais. A incorporação de conhecimentos andragógicos nesse processo ainda é uma lacuna a ser preenchida. Como forma de suprir essa demanda e promover a formação docente na graduação em Medicina, foram desenvolvidos produtos educacionais tecnológicos que orientam a elaboração de itens de prova em um processo de avaliação educacional formativa.
Relato de experiência: Este estudo teve como objetivo relatar essa experiência de trabalho colaborativo interdisciplinar, que envolveu educadores da área médica (especialistas do domínio), da linguística (especialistas da comunicação), da biblioteca (especialistas em ciência da informação) e do audiovisual (especialistas técnicos). Em consonância com os fundamentos da aprendizagem autodirigida da andragogia, os adultos aprendem melhor quando têm autonomia para seguir trajetórias personalizadas. Assim, criou-se uma trilha da aprendizagem, em que se descrevem os processos, as atividades e os agentes envolvidos. Criaram-se guias e roteiros, constituiu-se um comitê de avaliação, e produziram-se webinars, oficinas, uma plataforma digital para provas, videoanimações e um curso on-line. Os produtos educacionais foram incorporados ao processo de formação docente para orientar a elaboração dos itens de prova.
Discussão: Embora esses produtos possam favorecer a sistematização do processo avaliativo, a resistência à sua adoção e as limitações institucionais para sua implementação indicam que apenas a disponibilização de tecnologias não garante mudanças efetivas.
Conclusão: Ainda assim, os resultados apontam para avanços na conscientização dos docentes sobre a importância de uma avaliação formativa mais criteriosa e na construção de uma cultura avaliativa mais integrada.
Palavras-chave:
Educação Médica; Ensino Baseado em Competências; Avaliação Formativa; Itens de Prova; Interdisciplinaridade
INTRODUCTION
The number of medical schools has almost tripled in the last decade. Data from 2022 shows that there are 121 public and 268 private medical schools in Brazil1. With the increased number of vacancies, it has become necessary to professionalise medical teaching. Despite this demand, there is a reluctance on the part of the teaching profession to professionalise; teaching is under-appreciated in contrast to research; teaching is individualistic; there is a lack of stability in the teaching career; salary ceilings are flat and employment contracts are based on hours worked.
There are also challenges such as the transformation of curricula in health areas, their regulation by the Curriculum Guidelines and the adoption of adult learning methodology principles. This impacts the design of valid, reliable and viable assessment approaches that not only meet logistical and institutional constraints but can be used to assess the full spectrum of medical competencies.
Several studies have discussed how complex health education programmes are due to the need to prepare graduates for professional practice, with the alignment between curriculum and assessment. One mentions dental and medical schools that have implemented changes to their assessment systems in response to new national competencies and the need for more authentic assessment2. Another emphasises the importance of the perspective of all those involved in the process when designing assessment processes. It illustrates different assessment “states”, from no assessment to well-integrated systems in multiple dimensions of competence. It highlights the rapid growth of technology for developing and analysing assessment systems3. Another addresses the improvement of formative and summative assessment to monitor student progress with quantitative and qualitative information. The effectiveness of these programmes includes institutionalisation, centralisation, uniformity, comprehensiveness, coherence, continuity and integration. Its implementation requires political will, programme definition, the creation of an assessment centre, administrative and IT infrastructure4.
This is why we were motivated to think about how the systematisation of the assessment process in medical training can be improved through teacher training, mediated by technological tools. To comply with the andragogical principles5 in the assessment process, we envisage congruence between self-directed learning and formative assessment; previous adult experience as a learning resource in competency-based assessment; and learner motivation in constructive feedback.
The aim is to report on the experience of a collaborative interdisciplinary team in the development of technological educational products for the training of undergraduate medical teachers to guide the development of test items in a formative educational assessment process. Participants included medical professionals (domain specialists), linguists and librarians (communication and information specialists) and audiovisual technicians (technology specialists).
In addition to the principles of andragogy, in order for the products to be effective for adults, the guidelines of the Anísio Teixeira National Institute of Educational Studies and Research (INEP)6, which is linked to the Ministry of Education (MEC), are also followed for the preparation of test items. We also used Bloom’s Taxonomy as a basis for the achievement of educational objectives and student cognitive development7 and Miller’s Pyramid for structuring levels of competence in the assessment8.
These conceptual bases guided the construction of a learning path that was consistent with the context of the institution, the teachers and the students.
EXPERIENCE REPORT
To promote the training of undergraduate medical teachers in educational assessment, an interdisciplinary team was put together to design educational technology products that could guide and systematise the preparation of multiple-choice and discursive test items. The team was selected from among the institution’s own teaching and technical staff, based on competence, adherence to the project and availability; and by hiring external consultants. Collaborative work took place during weekly meetings for discussion, problematisation, ideation, research, planning and development of educational products.
The starting point was to propose a learning path, based on the scientific literature on educational evaluation and the institution’s assessment processes, for the development of guiding documents, such as guides and scripts, which could subsidise the development of educational products. The activities of this work programme, which took place between 2021 and 2023, can be seen in Table 1.
The learning path was structured in progressive stages, combining different resources to improve the development of test items. The process included introductory webinars, guides and workshops for developing assessment criteria, as well as the creation of an assessment committee. The implementation of the Easy Test Platform optimised the systematisation of the tests, while video animations and an online course expanded access to training content.
As can be seen in Table 1, each activity served as a prerequisite for the next and they all converged cumulatively to systematise, automate, optimise and monitor teaching practice with regard to writing test items.
Twenty-two teachers took part in Activity 1. The INEP/MEC criteria for preparing highly relevant exams were used for the structuring of the item in basic text, wording and alternatives (multiple-choice) or standard answers (discursive); and Bloom’s taxonomy was used to draft the wording in line with the learning objectives. The concept and characteristics of educational assessment, types of assessment and assessment instruments were discussed. This activity provided input for activities 2 and 3.
With this knowledge, the medical educators in the interdisciplinary team developed the Item Development Guide, customised to the institution’s practices. This material would serve as the basis for the assessment committee’s work. The time allocated to the committee’s review work is 25 hours a week between the reviewers. The criteria for selecting these reviewers were their engagement and interest in the training processes.
Activities 2 and 3 guided activity 4: in the light of the Guide, the committee began to review the test items prepared by the course lecturers before they were applied. During this period, they refined the criteria, generating complementary scripts for the Guide, such as the reference matrix, the checklist for checking test items and the standardised response document. Activities 2, 3 and 4, generated by activity 1, culminated in the incorporation of the “Easy Test Platform” for the inclusion of test items.
In activity 5, the platform was developed based on the requirements - modules, filters, functionalities, flow - proposed by the committee based on its experience of reviewing items. The use of the platform has not yet been evaluated quantitatively, but it has been validated qualitatively. In the committee meetings and in interactions with the teaching staff during the review, the perception of its use for the quality of the items and the organisation of the tests is self-declared positive.
Based on the platform, the guiding documents and systematised practices, activities 6 and 7 were proposed for the ongoing training of both the course lecturers and the assessment committee, as well as the initial training of lecturers new to the institution. The video animations and online course followed Knowles’ andragogical model5, based on the principles of the need to learn, the learner’s self-concept, the role of experiences, readiness to learn, orientation towards learning and motivation. The Guide was transformed into YouTube playlists of short video animations of up to five minutes, highlighting the guidelines for constructing the basic texts of the test items.
As activity 7, an online course was structured for a virtual learning environment, in which the video animations are inserted into didactic modules to systematise the main concepts involved in designing test items, such as, for example, the reference matrix, educational objectives (orders), the structure of the multiple-choice and essay test item (basic text, wording, alternatives/standard answer), and plausible distractors.
The design of each tool was not chosen arbitrarily but was based on didactic evidence observed by the team’s medical educators and aligned with the conceptual bases of andragogy and competency-based assessment.
RESULTS
The learning path generated results in terms of production. The use is experimental and has not yet been validated. The focus is on reporting on the processes of planning and developing the tools.
1. Webinars and online workshops: “Educational assessment and the preparation of written tests”:
The webinars took place synchronously, allowing for debate, and were recorded and made available for access asynchronously. They lasted 50 minutes and involved discussions about: educational assessment: cognitive and linguistic approaches, concepts and practices; test items; multiple choice and discursive tests. They are available at: https://youtube.com/playlist?list=PLuwvxOQt2Gn_2Aus0SSHflY2oMLJBbiK5&si=aUexkEQvQ70DwWjl.
2. Assessment Committee and the Easy Test Platform:
Made up of five teachers with a dedicated workload, the committee reviews the items. During the academic semester, there are regular exams for the basic and clinical cycles every six weeks and for the internship every twelve weeks. The course teachers prepare the test items in the light of the development guide, access the Easy Test platform and include them taking into account the semester of study, curricular component, educational strategy, topic and learning objectives. Once registered, the item goes through the two-level review flow. The committee conducts the first review by accessing the “My Tasks” tab to visualise the items and selects those it wishes to review. At the moment, only two reviewers conduct the second or final review, which is responsible for validating the item on the platform or not. If the item fails to meet the necessary requirements, it is returned to the teacher responsible for the registration, with comments and suggestions for improvement and resubmitted. Once validated, the item is available in the question bank, is no longer subject to editing and may be used in assessments.
To prepare the test, the teacher must organise the Assessment Model, inserting items available in the question bank. The assessment is scheduled on the platform by the course secretary and can be applied to students in print or online.
For teachers to learn how to use the platform, a six-month pilot was carried out on a spreadsheet containing the knowledge object, learning objective, skills to be tested and assessment criteria, which were filled in by the teachers on the committee. The platform developer then trained the committee, which passed on the instructions for use to the teachers.
Since then, the platform has been used by all the teachers, some of whom have had difficulties understanding how it works. In this case, the members of the committee clarify doubts, provide support and help with use.
Access to the platform is restricted to the institution’s teachers via https://unifae.provafacilnaweb.com.br/unifae. Images of the platform can be viewed at https://drive.google.com/drive/folders/13vPLso3P0xZZ9ou7ej4Cde5C1sQZksZA?usp=share_link.
3. Test item development guide:
The aim of the guide is to provide teachers with a methodology for preparing items so that they can produce tests of good technical quality. Teaching experience is of fundamental importance when developing items that are in line with the educational context.
4. Standard answer sheet:
To facilitate communication between test item reviewers and teachers, a standard answer document based on the guide has been developed and disseminated. As an example, here is a standard answer from the item improvement recommendations in standard answer document 6. “In order to improve the item, we ask you to review it, as it does not fulfil the order or the learning objectives for the stage of the course.”
5. Video animations:
In the video-animation playlists, the main concepts for the good construction of test items were summarised based on the guide and the standard answer document, and an inventory of information sources was generated for consultation and search for basic texts for the tests. The librarian listed types of information sources - scientific databases; university repositories or digital libraries; portals for scientific dissemination and scientific initiation projects; technical blogs; official government portals; highly relevant exam portals; social networks; teaching materials; case banks; question banks; image banks; and AI tools. The criteria for selecting the sources were to use only freely accessible sources in Portuguese. This data was organised in a table with the names of the information sources, the links where they can be accessed and the types of search engine strategies for each source.
For the videos on item construction and standard answers, scripts were drawn up, dividing up the content and describing the audio and image as guidelines for audiovisual production. Once the scripts were obtained for voice insertion, the video’s avatar character was created - a doctor - using the institution’s visual identity colours. For the information sources, the following criteria were established: no use of Boolean operators in the search examples to make the search easier; relate the information sources to the textual genres used as base texts (clinical cases, official documents; graphs; exams etc); classify by level of complexity: easy, medium, difficult; and provide search examples.
The medical educators reviewed and validated the finished scripts and videos:
Playlist 1: Test item development: Introduction to the Guide; Multiple choice; Discursive; and Standard answer(https://youtube.com/playlist?list=PLuwvxOQt2Gn8Q35_tLVFCiBB_qAeK8Qnt&si=_vxn4cqgi_VrVx_X).
Playlist 2: My library: Introduction to the platform; Platform resources; Clinical cases; and Orders (https://youtube.com/playlist?list=PLuwvxOQt2Gn9WA6jEqWXvGKJFOxFd0xyl&si=j5A1AtH6l6Qj0wMZ).
6. Online course: “Test item development”
The design of the course involved the creation of four didactic modules that address the main aspects of developing test items, from fundamental concepts in the field of educational assessment, such as reference matrix, competences, skills and assessment items, to the construction of multiple choice and discursive questions. Each module has the following structure: study materials, with texts to read and/or slides; video animations (activity 6, Box 1), to complement the readings; and assessment activities.
This is available at: https://classroom.google.com/c/Njc5NzgwOTg1MzU3 and access is restricted to teachers at the institution.
DISCUSSION
The webinars, workshops, video animations and guide sought to promote consideration of good practices in assessment with regard to the development of multiple-choice and discursive items.
With the creation of the assessment committee and the use of the Easy Test platform, the aim was to automate and systematise the previously manual and organic process. The platform provides monitoring and mapping of the assessments, organisation of the process, planning, timetabling and management of a question bank. This not only supports the logistics of the process but also promotes teacher training.
On the other hand, the standard answer document makes it easier for teachers to understand the demands for improving the items and affords reviewers a smoother process.
In addition, the flexibility of an online course allows teachers to access the platform at any time and from anywhere, adapting the training to their routines. The interactivity provided by video animations and practical activities keeps participants engaged and facilitates understanding of the content.
Educational products reflect cognitive progression through the levels of Bloom’s Taxonomy: (i) knowledge: video animations and introductory materials present fundamental concepts; (ii) understanding: commented examples help teachers to interpret good practice; (iii) application: workshops encourage the creation of test items; (iv) analysis: review tools enable different question formats to be compared and their validity to be discussed; (v) assessment: assessment committees and structured feedback enable adjustments and improvements to be made to teacher assessments; and (vi) creation: developing new test items consolidates learning.
They can also be mapped against the levels of Miller’s Pyramid: (i) know: guides and scripts help to understand assessment principles; (ii) know how: webinars and online courses demonstrate the application of assessment methodologies; (iii) show how: workshops allow teachers to practise developing test items; and (iii) do: implementing assessments on the platform, with iterative feedback, ensures that teachers actually apply the concepts.
As far as sustainability is concerned, products need a continuous updating model based on a modular and flexible design, so that new content can be included. The interdisciplinary team must remain active to review and update the materials on a regular basis. To facilitate and protect access, making them available in open digital repositories could be considered. Automated updating with the help of artificial intelligence could optimise the process.
Regarding its scalability to other medical schools, the model could be replicated as long as the technological resources are accessible and compatible with different institutional infrastructures. There needs to be flexibility in the curriculum and teachers need to be trained. Partnerships can be established and/or collaborative networks created so that different medical schools can share improvements and adaptations.
The products were designed with improvements in assessment quality in mind, efficiency gains in item creation and changes in teaching staff attitudes towards assessment. For the time being, what we have is the perception of improvement on the part of the assessment committee, teachers and students about the importance of more careful formative evaluation.
It is clear that quantitative data on the use of the platform, such as the number of test items reviewed, faculty engagement metrics, as well as a comparison between the quality of test items before and after the implementation of the guide and the platform, would better illustrate the workflow. Likewise, we could check the practical impact of video animations and online courses on daily teaching activities and also measure student performance by comparing grades before and after. However, these quantitative analyses are a future prospect for this work, which is still in progress.
Any integrative educational process, such as the one described here, is complex and brings challenges. Accordingly, just to illustrate the practical scope of the educational products for now, it can be asserted that the number of teachers involved with the Easy Test Platform is around 60. As regards the online course, although it has been finalised and is ready to be applied, it has yet to be implemented due to the change in the institution’s coordinator, which has led to a reorganisation of priorities.
CONCLUSION
In this experience report, the planning and development processes of the educational products were described and qualitatively recorded.
The main obstacles faced include: teachers’ unfamiliarity with educational technologies and resistance to change in the assessment culture; quality assurance and validation of materials; and sustainability and maintenance of the tools. Some strategies for overcoming this are continuous technical and pedagogical support from the interdisciplinary team and meetings for the active involvement of teachers.
As a future prospect, an evaluation study is planned to quantitatively investigate the impact of technological educational products on the development of teaching competences in educational assessment, as well as on the quality of student learning assessment. To this end, we intend to administer questionnaires to the students and conduct interviews with the teachers taking part in the training, ensuring compliance with all the required ethical aspects.
Our experience is that the change in assessment must be systematic and involve teachers and students. Bottom-up creativity is welcome, but it must be managed through governance structures2 and consider that implementing assessment systems is challenging and requires expertise3. It also corroborates related studies on the importance of a well-managed assessment system aligned with learning objectives2), (3), (4.
ACKNOWLEDGMENTS
Collaboration with librarian Glauce Teixeira Alves and audiovisual technician João Pedro Ferrari.
References
-
1 Scheffer M. Demografia médica no Brasil 2023. São Paulo: FMUSP, CFM, Cremesp; 2023 [acesso em 8 abr 2025]. Disponível em: Disponível em: https://amb.org.br/wp-content/uploads/2023/02/DemografiaMedica2023_8fev-1.pdf
» https://amb.org.br/wp-content/uploads/2023/02/DemografiaMedica2023_8fev-1.pdf -
2 Hays RB, Wilkinson T, Green-Thompson L, McCrorie P, Bollela V, Nadarajah VD, et al. Managing assessment during curriculum change: Ottawa Consensus Statement. Med Teach. 2024;46(7):874-84. doi: http://dx.doi.org/10.1080/0142159X.2024.2350522.
» https://doi.org/http://dx.doi.org/10.1080/0142159X.2024.2350522 -
3 Norcini J, Anderson MB, Bollela V, Burch V, Costa MJ, Duvivier R, et al. Consensus framework for good assessment. Med Teach . 2018; 40(11):1-8. doi: http://dx.doi.org/10.1080/0142159X.2018.1500016.
» https://doi.org/http://dx.doi.org/10.1080/0142159X.2018.1500016 -
4 Trocon LEA. Avaliação programática do estudante: estratégia institucional para melhor cumprir as funções da avaliação educacional. Rev Grad USP. 2016;1(1):53-8. doi: https://doi.org/10.11606/issn.2525-376X.v1i1p53-58.
» https://doi.org/https://doi.org/10.11606/issn.2525-376X.v1i1p53-58 - 5 Knowles MS, Holton EF, Swanson RA. The adult learner: the definitive classic in adult education and human resource development. 8th ed. London: Routledge; 2015.
-
6 Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. Guia de elaboração e revisão de itens. Brasília: Inep; 2010 [acesso em 7 maio 2025]. Disponível em: Disponível em: https://docs.ufpr.br/~aanjos/CE095/guia_elaboracao_revisao_itens_2012_INEP.pdf
» https://docs.ufpr.br/~aanjos/CE095/guia_elaboracao_revisao_itens_2012_INEP.pdf - 7 Bloom BS, editor. Taxonomy of educational objectives: the classification of educational goals. New York: David McKay; 1956.
-
8 Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(9): S63-7. doi: https://doi.org/10.1097/00001888-199009000-00045.
» https://doi.org/https://doi.org/10.1097/00001888-199009000-00045
Research data is available in the body of the document.
