Preparation and validation of a clinical empathy classification instrument: ICEC-MESABSTRACT
Introduction: Empathy is the ability to put yourself in the other’s shoes, understand and respond appropriately to their needs. In the clinical context, empathy is related to better therapeutic results and greater satisfaction for patients and health professionals. Demonstrated clinical empathy can present different dimensions of perception, ranging from simple emotional sharing to understanding the biomedical, psycho emotional and even social aspects of the patient.
Objective: To develop an instrument that allows the classification of clinical empathy demonstrated by medical students when using the Health Empathy Map (MES).
Methodology: This is a study with a mixed (qualitative and quantitative) and descriptive approach, developed in three stages. (1) Initially, an empathy classification model was proposed, which was the basis for the development of the clinical empathy classification instrument (ICEC-MES); (2) Assessment of the instrument’s content validity; (3) Evaluation of the instrument’s criterion validity and accuracy. To determine the level of agreement between the evaluating professors in relation to the classification of the student’s empathy in the proposed categories, the Fleiss Kappa index (p < 0.05) and the Gwet AC1 statistic were used when there was a significant imbalance and descriptive analysis of the results. data.
Results: The results showed that the instrument was able to identify the dimensions of clinical empathy, proved to be easy to understand and made it possible to classify the perception of empathy shown by the student. The concordance analysis revealed Gwet’s kappa/AC1 ranging from 0.872 to 0.209, with the highest indices being obtained in the ability to put oneself in the other’s shoes and the lowest indices in the affective dimension.
Final considerations: The study concluded that the tool presents evidence of reliability. The ICEC-MES makes it possible to identify and classify the clinical empathy demonstrated by the student, being considered, by the study participants, a useful tool to help the development of clinical empathy in care teaching settings.
Keywords:
Empathy; Classification; Medical students. Medical education
RESUMO
Introdução: A empatia é a capacidade de se colocar no lugar do outro, entender as necessidades dele e responder adequadamente a elas. No contexto clínico, a empatia está relacionada a melhores resultados terapêuticos e à maior satisfação de pacientes e profissionais da saúde. A empatia clínica demonstrada pode apresentar dimensões distintas de percepção, variando do simples compartilhamento emocional à compreensão dos apectos biomédicos, psicoemocionais e até mesmo sociais do paciente.
Objetivo: Este estudo teve como objetivo elaborar um instrumento que permita a classificação da empatia clínica demonstrada pelos estudantes de Medicina ao utilizarem o Mapa da Empatia em Saúde (MES).
Método: Trata-se de um estudo com abordagem mista (qualitativa e quantitativa) e descritiva, desenvolvido em três etapas: 1. inicialmente foi proposto um modelo de classificação da empatia que embasou a elaboração do instrumento de classificação da empatia clínica (ICEC-MES); 2. avaliação da validade de conteúdo do instrumento; 3. avaliação da validade de critério e precisão do instrumento. Para determinar o nível de concordância entre os professores avaliadores em relação à classificação da empatia do estudante nas categorias propostas, utilizaram-se índice kappa de Fleiss (p < 0,05) e a estatística AC1 de Gwet quando houve um desbalanceamento importante e a análise descritiva dos dados.
Resultado: Os resultados mostraram que o instrumento foi capaz de identificar as dimensões da empatia clínica, mostrou-se de fácil compreensão e possibilitou classificar a percepção da empatia demonstrada pelo estudante. A análise de concordância revelou kappa/AC1 de Gwet variando de 0,872 a 0,209, e os maiores índices foram obtidos na capacidade de se colocar no lugar do outro, e os menores índices, na dimensão afetiva.
Conclusão: O estudo permitiu concluir que o instrumento apresenta evidência de confiabilidade. O ICEC-MES possibilita identificar e classificar a empatia clínica demonstrada pelo estudante, sendo considerado, pelos participantes do estudo, uma ferramenta útil para auxiliar o desenvolvimento da empatia clínica em cenários de ensino assistenciais.
Palavras-chave:
Empatia; Classificação; Estudantes de Medicina; Educação Médica
INTRODUCTION
Modern Western medicine developed based on the scientific-positivist model, prioritising the biological dimension over the psychosocial. Distancing the physical from the abstract, that is, the feeling from the knowing, gaps have been created in medical knowledge, impacting negatively on the approach to subjective aspects of health and care1. This finding has triggered questions about medical training. There is now greater pressure on professionals to consider patients holistically, which means that doctors must be able to listen to and validate their patients’ feelings, opinions and points of view regarding their state of health and living conditions2.
In this sense, empathy emerges as a fundamental pillar of medical care, since establishing a good doctor-patient relationship is an essential factor in health care3. Clinical empathy can be understood as a doctor’s ability to understand the patient’s context, perspective and feelings; communicating this understanding and using it therapeutically4),(5.
Teaching medical students about the components of empathy and the importance of empathetic behaviour is therefore essential for medical professionalism6. According to Decety7 empathy has flexible components that are amenable to behavioural interventions.
Studies carried out by Batt-Rawden et al.8 and Moura et al.9 observed that educational interventions are effective in maintaining or increasing empathy. Various pedagogical strategies have been used to develop empathy in undergraduate medical courses, such as: the use of the arts; training in empathetic communication; interaction activities with real patients, virtual patients, actors and role-play; group or individual reflective activities on a medical consultation9.
However, characterising and quantifying empathy, taking into account that it is a subjective concept, has been a challenge. Of the tools used to quantify empathy in the context of clinical practice, the most widely used is the Jefferson Scale of Medical Empathy (JSPE), which has been adapted for students and has been validated in several countries. The JSPE predominantly assesses the cognitive domain of empathy10. Another instrument widely used to measure empathy in medical practice is the CARE (Consultation and Relational Empathy) scale. This is a scale applied after a consultation and is based on patients’ perceptions of the care they received from their healthcare professional11.
According to Ren, Min and Samarasekera12) because it has cognitive and emotional dimensions, the assessment of empathy requires the combination of instruments with different methodological approaches. The cognitive dimension is more easily measured by scales13) however, the affective dimension requires the use of qualitative instruments that have the potential to offer additional contributions to the findings of the scales14.
A tool designed to stimulate self-reflection and support the practice of empathy in health teaching scenarios is the Health Empathy Map (MES), which supports the conceptual appropriation of empathy by the practitioner and allows for a qualitative assessment of the student’s perception of the person being cared for14.
Sousa et al.14) conducted a study with 59 medical students which, in its qualitative phase, evaluated the characteristics of the reflections made by the students in the MES after reading clinical cases. The authors found that the students reflected on their own feelings if they were in the patient’s shoes, but in relation to the dimensions of empathy demonstrated in the reflections, a significant percentage only considered the biomedical aspects of the case, with few reflections on the psycho-emotional dimensions and social context.
This study highlighted the possibility of developing a tool that supports the classification of the empathic dimension perceived by students after completing the MES, with the aim of helping teachers in the process of giving feedback to students, favouring the development of empathy in the health professions. Therefore, the aim of this study was to develop and validate a tool to classify the dimensions of empathy shown by students when completing the MES.
METHODOLOGY
This is a descriptive study with a mixed approach (quantitative and qualitative) conducted in three phases (Figure 1).
Phase 1: Development of the tool
Initially, a model was drawn up for classifying the empathy shown by students when using the MES, based on the model empathy classification in the clinical care setting proposed in the study by Sousa et al.14 Three meetings were held in August 2021, in which the model was debated between the researchers until the final classification model was obtained (Table 1).
This model considers that empathy has two levels in terms of how it is manifested: 1. primary empathy (embodied resonance), an implicit form in which the emotions observed are felt and experienced by the observer; and 2. extended empathy, an explicit form in which the observer imagines what it would be like to be in the other person’s shoes, reflecting and making inferences, known as imaginary transposition, providing a higher level of social understanding15.
Considering that the practice of empathy in a healthcare context is supported by the ability to understand, beyond the biomedical dimension, the patient’s concerns, needs and emotional experiences, also taking into account the impact of the social context on their health and well-being, it was decided to include a system in the classification model (Table 1) that would allow the number of dimensions related to empathy identified by the student when completing the MES to be assessed.
After defining the model for classifying clinical empathy, the next step was to design an tool to be used to facilitate the classification of clinical empathy demonstrated in the MES (ICEC-MES). It was initially decided that the tool should have the following structure: The instructions are written at the top; just below them, a table containing three columns: a column containing the four questions in the MES (Q1, Q2, Q3 and Q4); a column describing the characteristics of the answers expected for each of the levels and dimensions of empathy shown by the student when filling in the MES; and another column for the score to be awarded when analysing the answers, which should consider the presence or absence of primary and extended empathy (yes or no) and the dimension of empathy perceived by the student: biomedical, affective and social, awarding 1 point for each of the dimensions identified.
Guidelines for interpreting the data are included below the table. The dimensionality of the empathy demonstrated will be ascribed by the sum of the points in Q2 and Q4 and the Q2/Q4 ratio will be used to assess the relationship between what the student identified of the patient’s needs (Q2) in relation to how much these needs were taken into account at the moment of help (Q4).
Phase 2: Assessment of the validity of the tool’s content
The first version of the tool was evaluated for content, semantics, relevance and feasibility. Following the recommendations of Barbour and Kitzinger16, we worked with a panel made up of eight experts, selected intentionally, who were familiar with the subject and had knowledge of the factors that affect empathy data. Four doctors (one geriatrician, one paediatrician, one family doctor and one endocrinologist), two educationalists, one nurse and one psychologist agreed to take part in the study. The meeting took place online via Google Meets and lasted two hours. The researchers invited them to take part in the study by e-mail or WhatsApp. All those who agreed to take part were sent a link to the informed consent form and a socio-demographic questionnaire containing: age, gender, undergraduate degree, length of time since graduation, length of time in clinical practice, length of experience as a teacher, area of work and qualifications.
The meeting began with a presentation of the project, clarification of the objectives of the study and the dynamics of the work. The eight guests were then given a brief summary of the definitions of empathy and its components. As the proposed instrument is based on the answers from the MES, a clean version of the MES and a completed version were presented to better illustrate how it is used. This was followed by a presentation of the classification model and the tool developed by the researchers in full and in parts for better visualisation. The meeting role play took place in two stages:
Stage 1 - Evaluation of the content of the tool: Each respondent, individually and without influence from the group answered questions related to semantics, content, relevance and the need to include or exclude items.
Stage 2 - Group discussion of the content of the items and assessment of relevance: Everyone returned to the plenary session for the debate, coordinated by the moderator, with the aim of collecting the comments made individually by the participants and verifying the consensus suggestions regarding the structure and semantic evaluation of the items of the tool. Rules of objectivity were adopted: speaking one person at a time and respecting the opinions of others in the group. The meeting went smoothly and clearly. All the suggestions made by the participants during the meeting were carefully noted down by a rapporteur and later analysed by the researchers, generating the second version of the tool.
In a second step, the second version of the tool obtained, after making the changes suggested by the panel of experts, was sent to four undergraduate medical teachers who work in clinical practice. Their suggestions for changes were discussed by the researchers and adjustments were made to the tool in an attempt to make it clearer and easier to use.
Phase 3: Assessment of the validity of the tool’s criteria and accuracy
Following the validity of the content, the tool’s criteria (effectiveness in predicting performance) and precision (ability to obtain identical results) were validated to ascertain whether the tool allows different assessors to agree on the final empathy shown by the students. We used a convenience sample made up of 40 students who should be in their fifth semester of medical school, have consented to take part in the study and signed the informed consent form (ICF).
This stage was executed in person, where after reading and signing the informed consent form, the students individually answered a sociodemographic questionnaire to characterise the group. Subsequently, the students watched a video recorded by an actor, the content of which was a detailed account of a patient’s health problems. The content of the video purposely addressed the three dimensions related to clinical empathy that would be assessed by the ICEC-MES: biomedical, affective and social. The case covered in the video had already been tested in terms of its dimensions by students who used the MES in the context of the tutorial group14. The students were then given the printed MES to fill in. The content of the students’ answers in the MES, as well as the sociodemographic questionnaires, were typed out in full into Excel spreadsheets to facilitate reading and evaluation.
In order to classify the students’ responses to the MES, using the ICEC-MES instrument proposed in the study, five undergraduate medical teachers were invited to take part and signed an informed consent form. The teachers received the invitation to participate, the ICF, a sociodemographic questionnaire and the spreadsheet with the students’ typed responses to the MES by e-mail. They were instructed to rate the empathy shown by the students using the ICE-MES developed by the researchers.
Ethical aspects of research
The study was duly approved by the Research Ethics Committee: Certificate of Submission for Ethical Appraisal (CAAE) No. 50985321.0.0000.514, opinion 4997089.
Data analysis
To characterise the sociodemographic data of the participants in all phases of the study, a descriptive analysis was conducted and percentages were presented as measures to describe the results of the variables studied. Fleiss’ Kappa index was used to analyse the agreement between the assessors, which is a measure that rates the agreement between three or more examiners for each of the evaluations on the Health Empathy Map completed by the students. For situations in which there was a very strong imbalance between the response possibilities, Gwet’s AC1 statistic was used. The two metrics range from -1 to +1, where a negative value indicates that the agreement between the assessors was lower than the agreement expected by chance. With -1 we are indicating that there was no agreement, 0 (zero) indicates that the agreement is no better than chance, and values greater than 0 (zero) represent an increasing agreement for the assessors, up to a maximum value of + 1, indicating perfect agreement17.
RESULTS
Table 2 describes the changes suggested by the raters and teachers in the ICEC-MES content validation stages. After finalising the content validation, the final version of the tool was obtained (Table 3).
NOTE: For the classification of the empathic dimension , the sum of the scores obtained in quadrants Q2 and Q4 should be considered, according to the table below:
Once the content had been validated, the accuracy of the tool was validated using the Fleiss Kappa/Gwet AC1 indices and descriptive analysis. The results of the review of agreement between the five assessors for classification of empathy, by quadrant of the ME, are described in table 4.
DISCUSSION
This study describes the stages in the development of a tool to classify the clinical empathy shown by students when completing the MES (ICEC-MES), taking into account the biomedical, affective and social dimensions. In general, the steps recommended in the literature were used with discretion and rigour. The final version of the tool includes the dimensions of clinical empathy labelled as pre-dimensional, one-dimensional, two-dimensional and multi-dimensional, according to the student’s ability to identify the patient’s biomedical, affective and social dimensions. In addition to the qualitative assessment, the tool provides a quantitative analysis, facilitating its use in teaching strategies and even in research.
All the stages in the development of the tool proved to be important in drawing up the final version. The first stage required extensive discussions by both the research team and health professionals with experience in the subject. This stage took into account aspects highlighted in the literature18 such as the definition of the content, the explanation of the cognitive processes to be assessed and the representation of each of the contents in the final model. The importance of the pre-test is noteworthy, as the changes made to the instrument led to a final version with a positive evaluation in terms of organisation, objectivity, clarity and understanding of the content. In addition, the ICEC-MES proved to be a tool that covers all the dimensions of clinical empathy addressed in the MES, and is representative of the universe of behaviours to be assessed.
The analysis of agreement between the assessors showed that the dispersion of responses between the quadrants of the MES was low for the category related to the student’s ability to put themselves in the other person’s shoes and moderate for the student’s perception of being touched by the patient’s story. The lowest level of inter-rater agreement was observed in the affective dimension, followed by the social dimension - when both the student’s perception of the patient’s needs and their behaviour in relation to the patient’s case were assessed. It was observed that these discrepancies were associated with the assessor’s speciality, since there was greater agreement between the evaluators from the geriatrics and paediatrics specialities, and less agreement with the pathology and psychopedagogy specialities. These results demonstrate that teachers had difficulty recognising and classifying the dimensions of empathy, which are fundamental to building a more humanised and patient-centred care practice. This skill faces significant obstacles when academic training remains anchored in traditional biomedical models, which emphasise technical and diagnostic aspects to the detriment of a holistic approach to the patient. The limitation of this perspective reinforces the urgent need for teacher development and the implementation of pedagogical strategies that broaden the view of comprehensive care, regardless of speciality. For truly effective training, it is essential that teachers are able to identify and stimulate reflection on the dimensions of empathy, promoting a more humanised approach to care3.
In view of this situation and taking into account the subjectivity of the issue, it is advisable to carry out prior training for the assessors in order to try to standardise observations with the aim of minimising discrepancies. However, further studies should be conducted to assess the effectiveness of this prior training on the accuracy of the analyses when using the tool.
This study makes important contributions to the field of education and research related to the development of empathy in the health professions. As far as we know, this is the first study to classify the dimensions of clinical empathy perceived by medical students when exposed to a clinical case. In general, research on the subject of empathy in the health sector has sought to measure its levels quantitatively, using self-report instruments, others have studied how the affective, cognitive and regulatory components of empathy are related and some have described qualitatively the perceptions of students when exposed to clinical care contexts. The ICEC-MES presents a simple methodology that is applicable in the various educational areas of healthcare, allowing educators to assess the dimensions that are being considered by their students when caring for patients or even in realistic simulation environments. Identifying the dimensions perceived by the students favours the construction of timely feedback that can contribute to self-reflection and the improvement of the conscious practice of clinical empathy.
Educational feedback plays a crucial role in the teaching and learning process, especially after the implementation of a new educational strategy. It not only informs students about their performance, but also directs their future endeavours, promoting more effective and meaningful learning(20) . Feedback acts as a bridge between the student’s learning and the educational objectives, encouraging critical reflection and increasing self-confidence. In addition, rapid feedback allows students to make adjustments in good time, maximising the impact of the strategy used.
Therefore, it is essential that teachers dedicate time and effort to providing adequate feedback to their students. This practice not only improves academic performance, but also strengthens the relationship between educators and students, creating a more collaborative and enriching learning environment19.
FINAL CONSIDERATIONS
The study concluded that the ICEC-MES shows evidence of reliability. The results allow us to infer that the ICEC-MES proved to be useful in guiding the teacher-assessor to identify and classify the dimensions of clinical empathy demonstrated by the student, and can contribute to supporting the feedback that should be given to the student, being considered by the study participants as a useful tool to help develop clinical empathy in healthcare teaching scenarios. This tool was devised and validated for use by teachers in the clinical practice learning scenario. Future studies could be conducted with the aim of making it possible for students to use it themselves, in a peer to peer assessment - stimulating their self-reflection on clinical practice, as well as in skills training.
Thus, the ICEC-MES is a tool that has the potential to be used in different assessment modalities: in formative assessment, by providing frequent feedback to students, helping to improve competences; in summative assessment, by being used to check whether the student has met the criteria required for certification; and in diagnostic assessment, by mapping skills and difficulties, allowing for more effective teaching planning.
References
-
1 Rios IC. Humanidades e medicina: razão e sensibilidade na formação médica. Cien Saude Colet. 2010;15(supl 1):1725-32 [acesso em 21 de setembro de 2022]. Disponível em: Disponível em: http://www.scielo.br/scielo.php?script=sci_arttext&pid=S1413-81232010000700084&lng=pt&tlng=pt
» http://www.scielo.br/scielo.php?script=sci_arttext&pid=S1413-81232010000700084&lng=pt&tlng=pt -
2 Larson EB. Clinical empathy as emotional labor in the patient-physician relationship. JAMA. 2005 Mar 2;293(9):1100-6 [acesso em 29 de agosto de 2021]. Disponível em: Disponível em: http://jama.jamanetwork.com/article.aspx?doi=10.1001/jama.293.9.1100
» http://jama.jamanetwork.com/article.aspx?doi=10.1001/jama.293.9.1100 -
3 Batista NA, Lessa SS. Aprendizagem da empatia na relação médico-paciente: um olhar qualitativo entre estudantes do internato de escolas médicas do Nordeste do Brasil. Rev Bras Educ Med. 2019;43(1 supl 1):349-56 [acesso em 15 de setembro de 2021]. Disponível em: Disponível em: http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0100-55022019000500349&tlng=pt
» http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0100-55022019000500349&tlng=pt -
4 Derksen F, Bensing J, Lagro-Janssen A. Effectiveness of empathy in general practice: a systematic review. British Journal of General Practice. 2013 Jan 1;63(606):e76-84 [acesso em 20 de agosto de 2021]. Disponível em: Disponível em: https://bjgp.org/lookup/doi/10.3399/bjgp13X660814
» https://bjgp.org/lookup/doi/10.3399/bjgp13X660814 -
5 Grosseman S, Novack DH, Duke P, Mennin S, Rosenzweig S, Davis TJ, et al. Residents’ and standardized patients’ perspectives on empathy: issues of agreement. Patient Educ Couns. 2014;96(1):22-8 [acesso em 10 de dezembro de 2021]. Disponível em: Disponível em: https://linkinghub.elsevier.com/retrieve/pii/S0738399114001451
» https://linkinghub.elsevier.com/retrieve/pii/S0738399114001451 -
6 Preusche I, Lamm C. Reflections on empathy in medical education: what can we learn from social neurosciences? Advances in Health Sciences Education. 2016 Mar 18;21(1):235-49 [acesso em 10 de outubro de 2022]. Disponível em: Disponível em: http://link.springer.com/10.1007/s10459-015-9581-5
» http://link.springer.com/10.1007/s10459-015-9581-5 -
7 Decety J, Cowell JM. Friends or foes. Perspectives on Psychological Science. 2014 Sept 17;9(5):525-37 [acesso em 10 de novembro de 2021]. Disponível em: Disponível em: http://journals.sagepub.com/doi/10.1177/1745691614545130
» http://journals.sagepub.com/doi/10.1177/1745691614545130 -
8 Batt-Rawden SA, Chisolm MS, Anton B, Flickinger TE. teaching empathy to medical students: an updated, systematic review. Acad Med. 2013;88(8):1171-7 [acesso em 15 de novembro de 2022]. Disponível em: https://journals.lww.com/academicmedicine/Fulltext/2013/08000/Teaching_Empathy_to_Medical_Students__An_Updated,.37.aspx
» https://journals.lww.com/academicmedicine/Fulltext/2013/08000/Teaching_Empathy_to_Medical_Students__An_Updated,.37.aspx -
9 Moura EP, Moura TP, Cordeiro JC, Chaves T de F, Peixoto AB, Peixoto JM. Estratégias atuais utilizadas para o ensino da empatia na graduação médica: revisão sistemática. Revista Eletrônica Acervo Saúde. 2021;13(2):e6374 [acesso em 15 de novembro de 2022]. Disponível em: Disponível em: https://acervomais.com.br/index.php/saude/article/view/6374
» https://acervomais.com.br/index.php/saude/article/view/6374 -
10 Hojat M, Mangione S, Nasca TJ, Gonnella JS, Magee M. Empathy scores in medical school and ratings of empathic behavior in residency training 3 years later. J Soc Psychol. 2005 Dec;145(6):663-72 [acesso em 10 de fevereiro de 2022]. Disponível em: Disponível em: http://www.tandfonline.com/doi/abs/10.3200/SOCP.145.6.663-672
» http://www.tandfonline.com/doi/abs/10.3200/SOCP.145.6.663-672 - 11 Scarpellini GR, Capellato G, Rizzatti FG, Silva GA da, Martinez JAB. Escala CARE de empatia: tradução para o português falado no Brasil e resultados iniciais de validação. Medicina (Ribeirao Preto Online). 2014;47(1):51-8.
-
12 Ren GSG, Min JTY, Ping YS, Shing LS, Win MTM, Chuan HS, et al. Complex and novel determinants of empathy change in medical students. Korean J Med Educ. 2016 Jan 27;28(1):67-78 [acesso em 10 de novembro de 2021]. Disponível em: Disponível em: http://kjme.kr/journal/view.php?doi=10.3946/kjme.2016.11
» http://kjme.kr/journal/view.php?doi=10.3946/kjme.2016.11 - 13 Ickes W, Marangoni C, García S. Studying empathic accuracy in a clinically relevant context. In: Ickes W, editor. Empathic accuracy. New York: Guilford Press; 1997. p. 282-310.
-
14 Sousa LU de R, Moura EP, Peixoto JM, Aredes J de S, Said C do C. Mapa da Empatia em Saúde como instrumento de reflexão em cenário de ensino não assistencial. Rev Bras Educ Med . 2021;45(4):e195 [acesso em 10 de novembro de 2022]. Disponível em: Disponível em: http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0100-55022021000400203&tlng=pt
» http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0100-55022021000400203&tlng=pt - 15 Fuchs T. Levels of empathy-primary, extended, and reiterated empathy. In: Lux V, Weigel S, editors. Empathy: epistemic problems and cultural-historical perspectives of a cross-disciplinary concept. Reino Unido: Palgrave Macmillan; 2017. p. 27-47.
- 16 Barbour R, Kitzinger J. Developing focus group research. London: Sage; 1999.
-
17 Fleiss JL, Levin B, Paik MC. Statistical methods for rates and proportions. New York: Wiley; 2003 [acesso em 10 de fevereiro de 2023]. Disponível em: Disponível em: https://onlinelibrary.wiley.com/doi/book/10.1002/0471445428
» https://onlinelibrary.wiley.com/doi/book/10.1002/0471445428 - 18 Pasquali L. Instrumentação psicológica: fundamentos e práticas. Porto Alegre: Artmed; 2010.
- 20 Hattie J, Timperley H. The power of feedback. Review of Educational Research. 2007;77(1):81-112.
Research data is available in the body of the document.
Source: Data from the study.