Evidence-based medicine competence at a public university: influence of graduation stageABSTRACT
Introduction: Evidence-Based Medicine (EBM) emerged in the 1990s to address the need for systematic clinical decision-making based on scientific research. In an era marked by an overload of information, EBM aimed to ensure that medical decisions are grounded in high-quality evidence. Although EBM has transformed clinical practice by introducing an objective and systematic approach, it has faced criticism for its tendency to undervalue the subjective aspects of the patient’s experience with illness. The integration of shared decision-making, formalized in 2001, seeks to incorporate patient preferences and values into the decision-making process without compromising the objective nature of EBM. This concept has gained prominence, promoting a more holistic and collaborative approach in medical practice. The incorporation of EBM into medical curricula remains a continuous challenge, often uncoordinated and addressed in a fragmented manner. Measuring competence in EBM is therefore crucial to identifying educational gaps and improving curricular interventions.
Objective: This study aims to assess competence in Evidence-Based Medicine among students at the Federal University of Juiz de Fora (UFJF) and identify factors associated with performance.
Method: A cross-sectional study was conducted with 767 UFJF students from the 1st to 9th semesters. Students were evaluated using the Fresno Test, which measures EBM skills. Variables such as academic semester, research experience, extracurricular EBM courses, and socioeconomic profile were analyzed. A descriptive data analysis was performed, along with comparisons using the Mann-Whitney and Kruskal-Wallis tests.
Results: 519 students participated, with Fresno Test scores ranging from 0 to 151, showing different medians across cycles (basic cycle: Md = 25, clinical cycle: Md = 59, clerkship: Md = 48). Performance improved throughout the course but slightly declined at the clerkship stage. Extracurricular courses and previous research experience were associated with better performance.
Conclusion: The study indicates improved competence in EBM over the course, with a slight decline in the final years. External interventions, such as EBM courses, significantly enhance student performance. Further research is needed to explore new assessment tools and expand the study.
Keywords:
Evidence-Based Medicine; Professional Competence; Medical School; Medical Education; Academic Performance
RESUMO
Introdução: A medicina baseada em evidências (MBE) surgiu na década de 1990 para sistematizar a tomada de decisão clínica com base na pesquisa científica. Em meio à exuberância de informações, a MBE passou a fundamentar as decisões médicas em evidências sólidas. Embora tenha transformado a prática clínica ao adotar um enfoque objetivo e sistemático, ela foi criticada por desconsiderar aspectos subjetivos da experiência do paciente. A introdução da decisão compartilhada, formalizada em 2001, integra as preferências e os valores do paciente ao processo decisório sem perder a objetividade, promovendo uma abordagem mais holística e colaborativa. A incorporação da MBE nos currículos médicos permanece um desafio, muitas vezes abordada de maneira fragmentada. Dessa forma, mensurar a competência em MBE é fundamental para identificar lacunas no ensino e aperfeiçoar as intervenções curriculares.
Objetivo: Este estudo tem como objetivos avaliar a competência em MBE entre os estudantes da Faculdade de Medicina da Universidade Federal de Juiz de Fora (UFJF) e identificar os fatores associados ao desempenho.
Método: Foi conduzido um estudo transversal com 767 alunos da UFJF, do primeiro ao nono período. Os alunos foram avaliados com o teste de Fresno, que mede habilidades em MBE. Analisaram-se as seguintes variáveis: período da graduação, experiência em pesquisa, cursos extracurriculares sobre MBE e perfil socioeconômico. Realizaram-se a análise descritiva dos dados e a comparação pelos testes de Mann-Whitney e Kruskal-Wallis.
Resultado: Participaram 519 alunos, com a pontuação no teste de Fresno variando de 0 a 151, com medianas diferentes entre ciclos (ciclo básico: Md = 25, ciclo clínico: Md = 59, internato: Md = 48). O desempenho melhorou ao longo do curso, mas caiu ligeiramente no internato. Cursos extracurriculares e experiência prévia em pesquisa foram associados a melhor desempenho.
Conclusão: O estudo mostra evolução na competência em MBE ao longo da graduação, com uma leve queda nos últimos anos. Intervenções externas, como cursos de MBE, melhoram significativamente o desempenho dos alunos. São necessários mais estudos para explorar novas ferramentas de avaliação e expandir a pesquisa.
Palavras-chave:
Medicina Baseada em Evidências; Competência Profissional; Faculdades de Medicina; Educação Médica; Desempenho Acadêmico
INTRODUCTION
Evidence-Based Medicine (EBM) emerged in the 1990s against a backdrop of rationalisation and systematisation of clinical decision-making, taking scientific research as a reference, at a time when a large volume of information is generated and only a portion of it can actually be considered high quality. This is why EBM consists of a series of skills and tools that professionals must use in their practice to ensure that patients have access to the best therapeutic options according to current evidence1.
The advent of EBM has made clinical decision-making objective and assertive, but its limitations have been criticised. The main issue is the undervaluing of subjective aspects of the patient’s relationship with the illness, which are equally important for clinical decision-making2. Despite these criticisms, the EBM movement has increasingly added the idea of shared decision-making to its practice, a concept formally introduced in 2001 and which is now widely disseminated, especially with regard to patient rights3. As a way of including patients in choices about their own health, shared decision-making aims to optimise treatments so that the parties agree on which therapeutic option will be most convenient, appropriate and effective4. Shared decision-making, therefore, does not limit or exclude the value of EBM, but rather highlights new perspectives on clinical decision-making, not just ascribing it to the objectivity of the professional, but to the will of the patient themselves.
With its expansion and gradual gain in influence, EBM currently goes beyond individual decision-making, as systematisation enables its use in guidelines, public policies and in guaranteeing a better standard of care5. The presence of EBM in multiple settings, from the doctor’s consulting room to health policies, makes its knowledge indispensable in undergraduate medical courses, when students intensify their consumption of scientific texts and are introduced to medical practice.
The inclusion of Evidence-Based Medicine in university curricula is still being debated because it is often not coordinated, with EBM-related subjects being introduced in different periods of the undergraduate course in a diluted manner. Given the particularities of each university and the lack of information in the literature about possible interventions, it is not yet possible to establish for course managers a standard for implementing EBM in the curriculum, and it is necessary for each place to recognise its own gaps in the teaching programme. Therefore, measuring this knowledge is fundamental to determining curricular interventions that make it more efficient to consolidate this knowledge throughout medical training6.
The aim of this study was to assess competence in Evidence-Based Medicine among students at the School of Medicine of the Federal University of Juiz de Fora and the factors associated with this performance.
METHODS
Design
This is a cross-sectional, analytical and census study. The students took the Fresno test, with each period of the course being assessed separately, in person, after the classes with the largest number of enrolled students. The tests were corrected and the scores analysed in conjunction with the variables collected.
Population
The target population of the study was all 767 undergraduate students regularly enrolled in the UFJF Medicine course between April and July 2023, from the first to the ninth semester. Clerkship students enrolled in later semesters were excluded due to the difficulty of meeting them in person.
Variables
The following independent variables were collected: semester of undergraduate study, which was related to the training cycle (1st to 4th basic cycle, 5th to 8th clinical cycle and 9th to clinical clerkship), previous experience in scientific production, extracurricular courses on EBM (both dichotomous, “yes” or “no” questions), interest and perception of the importance of scientific production and its practical application (in ordinal scale format) and socio-economic profile (according to ABEP 2022). The dependent variable was the score achieved in the Fresno test7),(8.
Instruments
The following instruments were used in the research: Self-designed questionnaire to collect the independent variables. Socio-economic questionnaire Brazilian Economic Classification Criteria 2022 (CCEB) prepared by the Brazilian Association of Research Companies (ABEP). This instrument has 15 questions related to assets, level of education and access to public services in the household. The Fresno Test, originally devised and validated in 2003 at the University of California7 and validated in Brazil8. The test assesses the first 3 steps of EBM, that is, asking, researching and critically evaluating through twelve short-answer open questions involving clinical scenarios, database research, mathematical calculations and critical and statistical evaluation of scientific articles. The questions are scored according to the correction guide7 and the test result is the sum of the scores with a maximum of 212.
Pilot
A pilot test was conducted before the data collection. The pilot group consisted of five sixth-year twelfth semester) medical students from the Federal University of Juiz de Fora in 2022. After applying the pilot test, the average time taken to solve the test was 35 minutes, as well as interpretation difficulties related to the research instruments.
Application procedure
The subjects with the highest number of students enrolled per semester were selected for the questionnaires. Prior to application of the questionnaires, a review of the respondents was conducted to ensure the avoidance of duplicates. The students were invited to take part and instructed about the time it would take to complete the questionnaires (approximately 40 minutes), the voluntary and anonymous nature of the research and to sign the ICF, and any questions they had were answered. After application of the questionnaire, the tests were distributed equally among the researchers involved and corrected using the correction guide in order to reduce the effect of subjectivity on the scores.
Statistical analysis
The data was stored in an Excel spreadsheet and analysed using the SPSS 26.0 programme. Exploratory analysis of the students’ socio-economic and academic profile was conducted and presented using absolute frequency (n) and percentage (%).
The Fresno test scores were analysed for normality using the Kolmogorov-Smirnov test. As the non-parametric distribution of these scores was accepted, the medians were compared using the Mann-Whitney and Kruskal-Wallis tests, according to the independent variables. The significance level (α) was set at 5% and the power of the test at 80%.
RESULTS
A total of 519 medical students from the Federal University of Juiz de Fora took part in the survey, which accounts for around 48% of all students at the school. At the time the questionnaires were administered, the respondents ranged from the first to ninth semester of the course. The semester with the highest number of participants was the first (15.6%) and the semester with the lowest number was the ninth (6.6%).
With regard to socio-economic indicators, the majority of students who responded to the survey are in class A (42.8%) and the smallest number are in class C2 (3.8%). No respondents belonged to classes D and E. We therefore presented this variable in the bivariate analysis in two groups: A or B and C.
As regards previous participation in scientific research groups and/or projects, 347 (66.9%) reported never having taken part in one before. Regarding the perceived importance of EBM for medical practice, 96% of the students attributed significant importance to it.
Only 9.1% of the students said they had already taken an external course about EBM.
The score obtained in the Fresno test ranged from 0 to 151, with a median of 40. The interquartile range found was 41.
Students in the basic cycle, from the first to fourth semesters, obtained a median (Md) of 25, with an interquartile range (IIQ) of 28. In the clinical cycle, from the fifth to eighth semesters, Md = 59 (IIQ = 41). In the clinical clerkship, represented in the study only by the ninth semester, Md = 48 (IIQ = 25).
According to the results presented by cycle, the cycle the student is in has an influence on their Fresno Test score when comparing the clinical cycle and the basic cycle (p < 0.001) and the clerkship and the basic cycle (p < 0.001). 0.001) and clerkship and basic cycle (p < 0.001), as well as having taken an extracurricular EBM course (p = 0.012) or having previous experience with scientific research (p < 0.001)
DISCUSSION
Better performance on the Fresno Test was found to be related to being in the clinical or clerkship cycle, having completed an EBM course and having previous experience in scientific research. In addition, a low level of adherence was evident among students at the end of their undergraduate studies, as well as a certain socio-economic homogeneity.
The majority of the study population was made up of medical students at the Federal University of Juiz de Fora who belonged to groups A or B in the ABEP socio-economic classification, most of whom had no previous research experience.
There are also disparities in the sample in terms of the greater participation of students from the initial periods when compared to the participation of students from the final years of the UFJF Medicine course. This pattern has also been observed in a similar study9 and may be related to undergraduate progression and its impact on students’ availability and willingness to participate in research after the first year of undergraduate study, as has been documented10.
The main variable analysed was the semester of study of the medical students, from the first to the ninth, grouped into cycles, basic, clinical and clerkship, and the influence this had on their competence in Evidence-Based Medicine. The medians found for the clinical cycle and basic cycle are close to those found in the Brazilian validation of the test for students who have had contact with EBM through epidemiology (Md = 55) and those who have not (Md = 18)8. There was a statistically significant improvement in student performance. This result is to be expected, given that at this stage of their degree students have contact both with clinical topics, which are important for understanding and contextualising the questions, and with statistics and epidemiology subjects, which provide the theoretical basis for most of the questions. In the clerkship programme, represented in this study by the ninth semester, the median was 48, corroborating the findings of another study9 which also observed a slight decline, although not statistically significant, in the competence in Evidence-Based Medicine of students in the final years of their degree when compared to those in the middle of the course. These results can be attributed to the curriculum with subjects related to EBM, including Epidemiology, Epidemiological Methods and Statistics, which are mainly taught in the clinical and basic cycles of the institution analysed.
Although these subjects contribute to the students’ training on the subject, the study also identified extracurricular courses focussed on Evidence-Based Medicine as a factor that improved student performance in the test. This result is in line with another study that compared performance on the Fresno Test before and after educational interventions on EBM11. The importance of these interventions is emphasised, as well as more targeted curricular activities, in order to reduce these disparities and guarantee the teaching of this fundamental knowledge universally within the regular undergraduate programme.
Previous experience with scientific projects was also a factor associated with better performance in the test, pointing to the importance of the research pillar in higher education as a way of consolidating the application of knowledge in Evidence-Based Medicine. This result adds to the other benefits of conducting such projects in medical education and training medical scientists, such as greater interest in academic careers, better academic performance, as well as serving as a gateway to research for students12),(13.
The Fresno Test was chosen for this study because at the time the project was devised it was the only one available and validated in Brazil. The questionnaire was answered in person to ensure better control over the circumstances in which it was applied. Currently, there are other tools for assessing knowledge in Evidence-Based Medicine with objective questions and shorter response times, such as the Assessing Competencies in Evidence-Based Medicine (ACE)14. For these reasons, such tools can make an important contribution to increasing student adherence and to the application process in future studies.
Originally, the Fresno Test was applied to groups representing the extremes of proficiency in the subject, which led to a limitation in the representation of intermediate proficiency groups for analysing and validating the tool7. In the case of a study whose population is made up of medical students, this may make the tool unsuitable.
In addition to these, other limitations were raised during our study as hypotheses for the low response rate on some of the proposed questions. These include the length of the questionnaire, which took up to 45 minutes to complete and included discursive answers that involved mathematical calculations, which may have contributed to the high number of unanswered questions. During application of the questionnaires, many doubts were also reported about the wording, which sometimes proved to be ambiguous or unclear, generating difficulties in interpreting the questions and thus possibly contributing to the participants’ low average performance.
CONCLUSION
Analysing the above data, it can be concluded that student competence in EBM evolves over the course of their undergraduate medical training. In addition, there may be a drop in this ability in the final years of the course, but further studies are needed to determine this possibility. Although improved competence in EBM is observed throughout the course, it can be seen that external training generates a significant improvement in student performance, making it clear that targeted interventions can have a positive impact on EBM teaching in medical courses. Finally, it is highlighted that new studies could be very beneficial for understanding the subject, since new, simpler tools for assessing competence in EBM are available, which could make it easier to expand the sample and conduct the study in different contexts.
References
-
1 Guyatt G. Evidence-based medicine. JAMA. 1992 Nov 4;268(17):2420-2425 doi: https://doi.org/10.1001/jama.1992.03490170092032
» https://doi.org/https://doi.org/10.1001/jama.1992.03490170092032 -
2 Freddi G, Romàn-Pumar JL. Evidence-based medicine: what it can and cannot do. Ann Ist Super Sanita. 2011;47(1):22-5. doi: https://doi.org/10.4415/ANN_11_01_06.
» https://doi.org/https://doi.org/10.4415/ANN_11_01_06 -
3 Djulbegovic B, Guyatt GH. Progress in evidence-based medicine: a quarter century on. Lancet. 2017 July 22;390(10092):415-23. doi: https://doi.org/10.1016/S0140-6736(16)31592-6.
» https://doi.org/https://doi.org/10.1016/S0140-6736(16)31592-6 - 4 Vidaeff AC, Turrentine MA, Belfort MA. Evidence based medicine - decades later. J Matern Fetal Neonatal Med. 2020 Feb;35(3):472-5.
- 5 Vere J, Gibson B. Evidence‐based medicine as science. J Eval Clin Pract. 2019 Dec;25(6):997-1002.
- 6 Maggio LA. Educating physicians in evidence based medicine: current practices and curricular strategies. Perspect Med Educ. 2016 Oct;5(6):358-61.
-
7 Ramos KD, Schafer S, Tracz SM. Validation of the Fresno test of competence in evidence based medicine. BMJ. 2003 Feb 8;326(7384):319-21. doi: https://doi.org/10.1136/bmj.326.7384.319.
» https://doi.org/https://doi.org/10.1136/bmj.326.7384.319 -
8 Salerno MR, Herrmann F, Debon LM, Soldatelli MD, Forte GC, Bastos MD, et al. Brazilian version of the Fresno test of competence in evidence-based medicine: a validation study. Scie Med. 2019 Feb;29(1): e32295. doi: https://revistaseletronicas.pucrs.br/scientiamedica/article/view/32295/17812
» https://revistaseletronicas.pucrs.br/scientiamedica/article/view/32295/17812 -
9 Smith AB, Semler LI, Rehman EA, Haddad ZG, Ahmadzadeh KL, Crellin SJ, et al. A cross-sectional study of medical student knowledge of evidence-based medicine as measured by the Fresno test of evidence-based medicine. J Emerg Med. 2016 May;50(5):759-64. doi: https://doi.org/10.1016/j.jemermed.2016.02.006.
» https://doi.org/https://doi.org/10.1016/j.jemermed.2016.02.006 -
10 Pereira M, Correia G, Severo M, Veríssimo AC, Ribeiro L. Portuguese medical students’ interest for science and research declines after freshman year. Healthcare. 2021 Oct;9(10): e1357. doi: https://doi.org/10.3390/healthcare9101357.
» https://doi.org/https://doi.org/10.3390/healthcare9101357 - 11 Lai NM, Teng CL. Competence in evidence-based medicine of senior medical students following a clinically integrated training programme. Hong Kong Med J. 2009 Oct;15(5):332-8.
-
12 Boyle SE, Cotton SC, Myint PK, Hold GL. The influence of early research experience in medical school on the decision to intercalate and future career in clinical academia: a questionnaire study. BMC Med Educ. 2017 Dec 11;17(1): e245. doi: https://doi.org/10.1186/s12909-017-1066-1.
» https://doi.org/https://doi.org/10.1186/s12909-017-1066-1 -
13 Solomon SS, Tom SC, Pichert J, Wasserman D, Powers AC. Impact of medical student research in the development of physician-scientists. J Investig Med. 2003 May;51(3):149-56. doi: https://doi.org/10.1136/jim-51-03-17.
» https://doi.org/https://doi.org/10.1136/jim-51-03-17 -
14 Maia FGS da S, Souza AKC de, Simas BCC de, Lopes IS, Bezerra MPRD, Diniz RVZ. Versão brasileira da ferramenta Assessing Competencies in Evidence-Based Medicine (ACE): um estudo de validação. Rev Bras Educ Med. 2022;46(3): e093. doi: https://doi.org/10.1590/1981-5271v46.3-20220081.
» https://doi.org/https://doi.org/10.1590/1981-5271v46.3-20220081
Research data is available in the body of the document
