ANALYSIS OF UNDERGRADUATE COURSE ASSESSMENT INSTRUMENTS (2006-2017) OF SINAES FROM THE PERSPECTIVE OF EVALUATION DICHOTOMIESAbstract
The National Higher Education Evaluation System (Sinaes), implemented in 2004, introduced significant advances to higher education by proposing an evaluation model anchored in three main pillars: institutional evaluation, evaluation of undergraduate programs, and evaluation of student performance. Since the system’s inception (specifically regarding the pillar focused on evaluating undergraduate programs), several instruments have been adopted to support the acts of course authorization, recognition, and renewal of recognition in both on-campus and distance-learning formats. In this context, the present article aims to analyze the undergraduate program evaluation instruments instituted by the National Institute for Educational Studies and Research Anísio Teixeira (Inep) between 2006 and 2017, mapping the changes that have occurred within the underlying concepts and evaluation dichotomies: internal or external, formative or summative, quantitative or qualitative, and integrated or point-in-time. Methodologically, this is a descriptive study based on documentary research. The expected outcome is to contribute to improving Sinaes, chiefly by providing Inep with inputs to refine the evaluation instruments that underpin the acts of course authorization, recognition, and renewal of recognition.
Keywords:
evaluation of higher education; National Higher Education Assessment System; assessment instruments.
Resumo
O Sistema Nacional de Avaliação da Educação Superior (Sinaes) foi implementado em 2004 e trouxe avanços significativos para a educação superior, propondo uma avaliação ancorada em três pilares principais: a avaliação institucional, a avaliação de cursos de graduação e a avaliação do desempenho dos estudantes. Desde a implementação do Sistema, especificamente em relação ao pilar focado na avaliação de cursos de graduação, vários instrumentos foram implementados, para embasar os atos de autorização, de reconhecimento e de renovação de reconhecimento de cursos, nas modalidades presencial e à distância. Nesse contexto, este artigo propõe, como objetivo geral, analisar os instrumentos de avaliação dos cursos de graduação instituídos pelo Instituto Nacional de Estudos Educacionais Anísio Teixeira (Inep), no período compreendido entre 2006 e 2017, buscando mapear as mudanças ocorridas dentro das concepções e das dicotomias de avaliação: interna ou externa, formativa ou somativa, quantitativa ou qualitativa e integrada ou pontual. No que se refere aos aspectos metodológicos, se realizou uma pesquisa descritiva e, quanto aos procedimentos, foi feita uma análise com base em pesquisa documental. Espera-se, como resultado, que este artigo traga contribuições para o aprimoramento do Sinaes, principalmente no que tange a fornecer subsídios para o Inep para o aperfeiçoamento dos instrumentos de avaliação utilizados para subsidiar os atos de autorização, de reconhecimento e de renovação de reconhecimento de cursos.
Palavras-chave:
avaliação da educação superior; Sistema Nacional de Avaliação da Educação Superior; instrumentos de avaliação.
Resumen
El Sistema Nacional de Evaluación de la Educación Superior (Sinaes) fue implementado en 2004 y trajo avances significativos para la educación superior, proponiendo una evaluación basada en tres pilares principales: la evaluación institucional, la evaluación de los cursos de grado y la evaluación del desempeño estudiantil. Desde la implementación del Sistema, específicamente en relación con el pilar enfocado en la evaluación de los cursos de grado, se han implementado varios instrumentos para fundamentar los actos de autorización, reconocimiento y renovación del reconocimiento de cursos, tanto en la modalidad presencial como a distancia. En este contexto, este artículo propone, como objetivo general, analizar los instrumentos de evaluación de los cursos de grado instituidos por el Instituto Nacional de Estudios e Investigaciones Educativas Anísio Teixeira (Inep) en el período comprendido entre 2006 y 2017, buscando mapear los cambios ocurridos dentro de las concepciones y dicotomías de evaluación: interna o externa, formativa o sumativa, cuantitativa o cualitativa e integrada o puntual. En cuanto a los aspectos metodológicos, se realizó una investigación descriptiva y, respecto a los procedimientos, se llevó a cabo un análisis basado en investigación documental. Se espera, como resultado, que este artículo aporte contribuciones para el perfeccionamiento del Sinaes, principalmente en lo que respecta a proporcionar insumos para el Inep en el mejoramiento de los instrumentos de evaluación utilizados para sustentar los actos de autorización, reconocimiento y renovación del reconocimiento de cursos.
Palavras clave:
evaluación de la educación superior; Sistema Nacional de Evaluación de la Educación Superior; instrumentos de evaluación.
1 INTRODUCTION
In 2024, the National Higher Education Evaluation System (Sinaes), established in Brazil by Law No. 10,861 of 14 April 2004, celebrated its twentieth anniversary.
Sinaes was created in line with the provisions of Law No. 9,394 of 20 December 1996, which sets out the National Education Guidelines and Framework (LDB), and Law No. 10,172 of 9 January 2001, which instituted the first National Education Plan (PNE) after the LDB. Article 4 of the PNE stipulated that the Federal Government should establish a National Evaluation System. It is also important to note that implementing Sinaes met one of the higher-education proposals included in the 2002 presidential campaign platform of Luiz Inácio Lula da Silva, which called for a review of the evaluation system then in force, as Ristoff and Giolo (2006, p. 194) point out:
Proposal 12:
Review the current evaluation system, which includes the National Course Examination - ENC, or Provão - and implement a national system of institutional evaluation, drawing, among other sources, on the experience of the Institutional Evaluation Programme of Brazilian Universities (PAIUB).
When it was created, the law that instituted Sinaes defined its purpose as follows:
[...] improving the quality of higher education, guiding the expansion of its provision, permanently increasing its institutional effectiveness and academic and social impact, and, above all, deepening the social commitments and responsibilities of higher-education institutions by valuing their public mission, promoting democratic values, respecting difference and diversity, and affirming institutional autonomy and identity.” (Brasil, 2004b)
From this perspective, as an integrated evaluation system, the legislation that created Sinaes emphasized the formative and public character of evaluation, as well as respect for diversity and the social responsibility of higher education institutions (HEIs) - whether universities, university centers, standalone colleges, Federal Institutes of Education, Science and Technology (IFs), or Federal Centers for Technological Education (CEFETs) - seeking an integration of instruments addressing quantitative and qualitative aspects related to three pillars: 1) institutional evaluation, encompassing external evaluation and internal evaluation, the latter managed by each institution’s Internal Evaluation Committee (CPA); 2) course evaluation; and 3) student performance evaluation, institutionalized through the National Student Performance Examination (Enade).
In all three evaluation pillars - institutional evaluation, undergraduate course evaluation and Enade - results are expressed on a five-point scale (five being the highest and one the lowest), assigned both to each individual dimension and to the overall set of dimensions assessed.
Since Sinaes was institutionalized, the National Institute for Educational Studies and Research Anísio Teixeira (Inep), with the approval of the National Commission for the Evaluation of Higher Education (Conaes), has adopted several undergraduate course evaluation instruments to support the regulatory acts of course authorization, recognition and renewal of recognition in both on-campus and distance-learning modalities. The first instrument was introduced in 2006, followed by new versions in 2008, 2010, 2012, 2015 and 2017.
This study focuses specifically on Sinaes’s second pillar - undergraduate course evaluation - which involves on-site verification by experts appointed by Inep, of the higher-education conditions offered to students. The evaluation instrument covers three dimensions: (i) the course’s didactic-pedagogical organization; (ii) the teaching and tutorial staff; and (iii) infrastructure.
Accordingly, the general objective of this article is to analyze the undergraduate course evaluation instruments issued by Inep between 2006 and 2017, mapping the changes in underlying concepts and in the following evaluation dichotomies: formative or summative, quantitative or qualitative, internal or external, and integrated or oneoff.
Regarding methodological aspects, a descriptive study was carried out to provide an in-depth view of higher-education evaluation. The procedures involved documentary research, entailing detailed examination of the assessment instruments implemented between 2006 and 2017. This approach not only allowed us to identify and document the essential characteristics of the object under study, but also offered a detailed analysis that helps clarify the nuances and complexities of the evaluation dichotomies.
2 EVALUATION DICHOTOMIES
In recent years, it’s been observed that activities related to evaluation practices have intensified both in Brazil and around the world. This occurs in the corporate context (involving production processes, personnel selection and quality certifications, for instance) and in areas linked to academic training, including teaching, research and extension courses. Leite (2015) extends this idea, stressing that we are living in times of evaluation in every sector and segment of human activity. Complementing this, Belloni, Magalhães and Sousa (2001, p. 14) reinforce that:
[...] assessment is an everyday, spontaneous action carried out by any individual in relation to any human activity; it is, thus, a fundamental instrument for knowing, understanding, improving and guiding the actions of individuals or groups.
Although it did not originate in educational institutions, evaluation was widely incorporated into their practices, as Dias Sobrinho (1997, p. 20) points out:
Of all social institutions with public functions - the training function is undeniably public - the university is perhaps the one most inclined toward evaluation, even though it does not necessarily do so in the global and integrated manner proposed by institutional evaluation. This is a relatively recent modality in the world, and even more so in Brazil. Different forms of evaluation have always been carried out in educational institutions and particularly in universities.
Within education, as Leite (2005) notes, two formats of evaluation stand out most clearly. The first relates to what the author calls “educational or learning evaluation” (Leite, 2005, p. 33), which focuses on the learning of individuals or groups aimed at the acquisition of competences (encompassing knowledge, skills and attitudes); in short, this evaluation centres on analysing the performance of individuals or groups and may also involve curriculum evaluation. The second format corresponds to what the author calls “institutional or public-policy evaluation” (Leite, 2005, p. 33), which seeks to evaluate institutions as a whole or the public policies, encompassing programmes, plans and projects in their global and contextualised dimensions.
In both formats, evaluation seeks a better understanding of the strengths and weaknesses of the phenomena under scrutiny, so that faults and errors can be reviewed and priorities defined for more democratic management, whether at the institutional or global level.
On the basis of these assumptions, Belloni, Magalhães and Sousa (2001) state that evaluation is a systematic process of analysing a given activity or fact, with a view to better understanding all its dimensions and implications, aiming at its improvement. Leite (2005, p. 15) emphasises that:
evaluation is neither a neutral nor a universal act. One cannot technically evaluate an institution such as the university without thereby invoking a political and philosophical presupposition about conceptions of the world and of society.
Ristoff (2024, p. 83) likewise argues that “to evaluate is therefore not to verify; it is, above all, to carry out rigorous analyses, establish correlations, contextualise and produce inferences and judgements duly grounded in the observed reality.”
Dias Sobrinho (2003) contends that evaluation should be understood as a social practice that involves multiple dimensions - ethical, political and epistemological. For the author, evaluation should contribute to human and social development by promoting the improvement of educational processes, the enhancement of quality and the democratisation of teaching.
Against this background of evaluation concepts, Ristoff (2024) stresses that a set of dichotomies informs the principles, objectives and methodologies employed in evaluative processes, the best-known being: i) internal evaluation or external evaluation; ii) formative evaluation or summative evaluation; iii) quantitative evaluation or qualitative evaluation; and iv) integrated evaluation or point-in-time evaluation. The author also notes that other dichotomies featured in the discussions of the Special Evaluation Commission (CEA), responsible for coordinating the review and reorganisation of the evaluation system that gave rise to Sinaes, but that the four highlighted in this article demanded the most debate and contributed most to the construction of the evaluation system implemented from 2004 onward.
With regard to the dichotomy between internal and external evaluation, Dias Sobrinho (2003) works on the premise that the former must involve the active participation of the internal community, which is responsible for establishing the principles, objects of evaluation, subjects, procedures and uses of the evaluative process. Dias Sobrinho (2000) further states that in internal evaluation, which takes place through self-evaluation, the evaluators are not only subjects but also form part of the object to be evaluated.
The aim of this evaluation is to give the academic community a better understanding of the institution, the courses it offers, and the programmes and projects it makes available to society. In this context, all data and information obtained through self-evaluation must be consolidated in official reports that have to be discussed and approved by the academic community.
External evaluation, in turn, confirms the transparency and public nature of evaluation through the effective participation of external members of the scientific community, public institutions linked to education, science and technology, and representatives of organised civil society (Dias Sobrinho, 2000). In the external evaluation of undergraduate courses, the evaluators are faculty members from highereducation institutions (active or retired) holding a master’s or doctoral degree, who belong to the National Higher Education Evaluation System’s Bank of Evaluators (BASis), managed by Inep.
External evaluation, as an examination from the outside in, can be carried out with a view to analysing and correcting any errors of perception produced by internal evaluation, given that the latter may, in some cases, be accustomed to the routines and evaluative processes of an institution or course (Dias Sobrinho, 2003).
Dias Sobrinho (2000) reinforces the idea that internal evaluation provides the foundations for external evaluation, and that external evaluation, in turn, supplies inputs for the execution of new internal evaluative processes.
It is also important to highlight, in relation to this dichotomy, what is stated in the foundational document that led to Sinaes:
[...] the combined actions of internal and external evaluation are important processes of discussion and reflection on major themes of pedagogical, scientific and technological policy, as well as for decision-making, seeking to strengthen or redirect actions” (Brasil, 2004a, p. 105).
As for the second dichotomy, concerning formative evaluation or summative evaluation, it can be said that formative evaluation operates on the premise that the primary character of evaluation should be educational, taking into account different opinions and positions in interpretation and interpersonal relationships (Dias Sobrinho, 2000). In this sense, such evaluation tends to be more participatory.
For Ristoff (2024), formative evaluation strategies focus on processes, with a predominant use of qualitative methods. This evaluation tends to be descriptive and conducts detailed analyses of the strengths and weaknesses of the object under evaluation. On this basis, its fundamental principle is to ascertain what is and is not functioning well, always listening to those directly involved.
Summative evaluation, on the other hand, is carried out to enable decisionmaking about the object being evaluated, such as a course, ultimately expressing a verdict on its functioning (Ristoff, 2024), whether in terms of access to the highereducation system (authorisation), continued presence in the system (recognition and renewal of recognition), or closure of its academic activities within that system. It is thus used as a mechanism for informing society about the legal status of a course’s operation. Summative evaluation generally relies predominantly on quantitative methods and focuses more on results.
Addressing the difference between the two forms of evaluation, Ristoff (2024) stresses that formative evaluation seeks information for the object evaluated so that those involved can improve methods, routines and processes, whereas summative evaluation seeks information about the object evaluated so that external agents can make decisions regarding a course’s authorisation, recognition or renewal of recognition, as well as an institution’s accreditation or re-accreditation.
In the third dichotomy, which involves qualitative evaluation or quantitative evaluation, Ristoff (2024, p. 60) notes that
[...] the evaluations implemented in Brazil in the last decades of the 1900s were inspired either by the Anglo-American model - basically quantitative, classificatory and ranking-oriented - or by the French and Dutch model - predominantly qualitative, analytical and institutional.
The purpose of evaluation should not be limited to the collection of information, measurement and quantification of indicators or dimensions of analysis of a course or institution - features that lie at the core of a more quantitative approach to evaluation. Evaluation demands a more complex outlook, one that also contextualises the object being evaluated - such as a course or institution - understanding its effects on and within society, while respecting its diversity and multiple functions; elements which characterise qualitative evaluation. In evaluation studies, it is common to understand that whereas quantitative evaluation tends to be more objective, qualitative evaluation tends to be more subjective, requiring a more refined look at what will be evaluated.
At the genesis of Sinaes, the CEA emphasised that, in internal evaluation or selfevaluation, instruments could make greater use of qualitative methods, whereas external evaluation instruments would have to rely more on quantification, although they could include open-ended questions and spaces for evaluators’ observations and comments. It was also considered that, to designate a method as quantitative or qualitative, one should look at predominance, not exclusivity, of quantitative measurements or of descriptions and interpretations (Ristoff, 2024).
In the sphere of quantitative and qualitative evaluations, it is also important to underscore what Dias Sobrinho (2003, p. 45) states:
[...] privileging the qualitative, given its greater educational potential, doesn’t mean denying the quantitative nor establishing any opposition between them. Statistics, databases and quantitative studies form a solid basis for evaluation. However, this approach is insufficient and incapable of capturing the full richness of meanings if reflections and meaning-making procedures of a qualitative nature are not applied to it.
The foundational document for the creation of Sinaes, drafted by the CEA, also addresses the quantitative-qualitative evaluation dichotomy, emphasising that:
The complexity of higher education, both at the institutional level and at the system level, requires the use of multiple instruments and the combination of diverse methodologies. For example, it is no longer appropriate to debate the false aporias of the quantitative and the qualitative or of the objective and the subjective, but rather to employ the various instruments and distinct methodological perspectives in a combined, complementary manner and in accordance with the needs of analysis and judgement (Brasil, 2004a, p. 83).
In the fourth dichotomy, concerning integral evaluation or point-in-time evaluation, the logic is that evaluative processes should seek a global analysis of evaluation (integral evaluation) and not merely focus on isolated evaluation instruments that don’t interact with one another (point-in-time evaluation).
In this regard, Ristoff (2024) raises the concern that isolated procedures, often lacking articulation with a coherent programme or set of policies, may be taken as the sole evaluative possibility and that their results may be disseminated as an established truth. The author underscores that, before the implementation of Sinaes, the evaluation carried out by the Ministry of Education (MEC) “was fragmented, topical, producing distorted images of institutional complexity and of the courses, viewed and evaluated on the basis of partial, incomplete, and unreliable data” (Ristoff, 2024, p. 85). He also states that these isolated procedures were “pseudo-evaluations or quasi-evaluations and not true evaluations, because they did not address, or addressed only tangentially, value-laden issues” (Ristoff, 2024, p. 85).
Beyond the integration of evaluation instruments, integrated evaluation also foresees a direct relationship with information tools such as the Higher Education Census and the Registry of Higher-Education Institutions. Discussing integrated evaluation, Dias Sobrinho (2000) argues that emphasis should be placed on teaching, research and outreach processes, as well as on management and infrastructure, whenever possible in an integrated fashion, taking as a reference and explanatory dimension the aims of the object of evaluation. One must proceed on the premise that to integrate implies interpreting the causes, consequences, contexts and processes of the phenomenon one seeks to understand, bearing in mind that the quality of what is being evaluated resides more in the whole than in isolated aspects. In this context:
The construction of a higher-education evaluation system thus depended on a way of seeing the world of higher education, it meant understanding it as a whole, composed of countless parts and embedded in an educational, socioeconomic, cultural and political context (Ristoff, 2024, p. 90).
3 THE UNDERGRADUATE-COURSE ASSESSMENT INSTRUMENTS
In the sphere of Brazilian higher education after-LDB, on-site assessment instruments for undergraduate courses were created, at first, within the Ministry of Education’s Secretariat for Higher Education (SESu) and the Secretariat for Secondary and Technological Education (SEMTEC), later renamed the Secretariat for Professional and Technological Education (SETEC) of MEC. Whereas the former was responsible for assessing bachelor’s and licentiate degree courses, the latter assessed highereducation technology courses. In addition, for courses offered via distance education (EaD), there was liaison with the Secretariat for Distance Education (SEED).
From 2002 onward, by virtue of MEC Ordinance No. 990 of 2 April, Inep took the lead in evaluating undergraduate courses in Brazil and began the Assessment of Supply Conditions (ACO), later renamed Assessment of Teaching Conditions (ACE). The chief purpose of this assessment was to verify, on site, through a commission of specialists appointed by the MEC, the quality of undergraduate courses, focusing on three dimensions: “didactic-pedagogical organisation” (dimension 1), “teaching staff” (dimension 2) and “infrastructure” (dimension 3).
These evaluation processes also served the regulatory function in higher education, that is, the procedures that culminated in the acts authorising a course (authorisation, recognition or renewal of recognition). At the end of each assessment, the commission of specialists assigned ratings for each of the dimensions assessed, as well as an overall rating for the course, on the following scale: Very Good Conditions (CMB), Good Conditions (CB), Regular Conditions (CR) and Insufficient Conditions (CI).
With the implementation of Sinaes in 2004, the Undergraduate-Course Assessment (ACG) came into effect, replacing the ACE, while operational responsibility for evaluations remained with Inep. According to the law that instituted Sinaes:
Art. 4 The evaluation of undergraduate courses aims to identify the teaching conditions offered to students, particularly those related to the teaching staff profile, physical facilities and didactic-pedagogical organisation.
§ 1 The evaluation of undergraduate courses shall make use of diverse procedures and instruments, among which, obligatorily, external evaluation by commissions of specialists in the respective fields of knowledge (Brasil, 2004b, 2022).
Within Sinaes, in the pillar concerning course evaluation, the dimensions “didactic-pedagogical organisation” (dimension 1) and “infrastructure” (dimension 3) were retained, and there was a slight change in the dimension related to staff, which at first was called “teaching, students and technical-administrative staff” and later came to include the assessment of tutors - present in distance courses or in on-campus courses that have part of their workload delivered via EaD - so that, in the most recent instruments, this dimension was renamed “teaching and tutorial staff” (dimension 2).
Since its inception until today, and despite efforts to preserve the system’s essence, a series of changes have been made, chiefly to the assessment instruments designed both for the external evaluation of institutions and for the evaluation of their undergraduate courses.
Focusing specifically on the undergraduate-course assessment instruments - the object of the present analysis - the following table shows the changes introduced from the creation of Sinaes to the present day:
Pursuant to legislation, every undergraduate-course assessment instrument implemented from 2006 onward was organised around the three Sinaes dimensions: “didactic-pedagogical organisation”, “teaching and tutorial staff” and “infrastructure”. For each dimension, a variety of indicators were set out, evaluated on the basis of predefined analysis criteria contained in the respective instruments. The following table shows the number of indicators making up each evaluative dimension in the instruments adopted from 2006 to 2017.
Up to 2017, the analysis criteria within each indicator were rated on ascending scales of quality excellence, from 1 to 5, as follows: 1 (does not exist), 2 (insufficient), 3 (sufficient - minimum quality criterion), 4 (very good/very well) and 5 (excellent). If the indicator did not apply to the course’s reality, on legally grounded grounds, the rubric Not Applicable (NA) had to be assigned. Moreover, for a given set of inter-related indicators, the evaluation commission also filled in a mandatory overall narrative for the analysis category, and likewise for the dimension as a whole. According to Inep (2015b, p. 37):
In every dimension there is space for the evaluator’s narrative, which highlights the qualitative aspect of the assessment. Thus, it is through the scales and evaluators’ descriptions-based on on-site observations and analysis of documents, that the objective and subjective aspects of assessment communicate.
These instruments also contained a category entitled “Legal or Normative Requirements”, under which the commission had to analyse a set of items and state whether they were met by the course, or not, or were inapplicable, on legal grounds.
From 2017 onward, in addition to structuring the instruments in the three Sinaes dimensions, each comprising a set of indicators, the analysis criteria for assigning ratings came to be based on attributes, understood as the smallest part of an analysis criterion and representing an element that must be verified during assessment. Thus, to award ratings 1 to 5, each indicator requires a set of attributes.
Inep (2018, p. 14), clarifying the notion of attributes in the logic of the new instruments published from 2017, stresses that:
There are cases in which the difference between the analysis criterion for each rating lies in the number of associated attributes. In others, the variation between analysis criteria is characterised by the quality of the attribute. Thus, given that rating 3 satisfactorily characterises the object evaluated, the analysis criteria for ratings 4 and 5 were built on an additive logic.
Following the same logic, in the opposite sense, the analysis criteria for ratings 2 and 1 do not fully encompass the attributes of rating 3.
According to Inep (2015b, p. 36), during discussions on changes to the assessment instruments over the lifespan of Sinaes, certain issues were diagnosed in the indicators and analysis criteria, such as:
[...] non-uniform titles, composite criteria, differing scales, unaddressed logical possibilities, highlight indicators, distinct indicators, essential indicators, differentiated scales, subjective descriptors, rounding in relation to the overall rating, among others.
In 2017, the changes to the instruments brought new elements for discussion, including:
[...] pertinent legislation; recently enacted legislation; statistical studies; analysis of demands from organised civil society and the academic community; PNE targets; dialogue with members of the Sinaes National Bank of Evaluators - BASis; and ongoing dialogue between Inep and Seres, which uses the instruments in its regulatory practice (Inep, 2017b, p. 7).
4 ANALYSIS OF THE UNDERGRADUATE-COURSE ASSESSMENT INSTRUMENTS IN THE LIGHT OF THE EVALUATION DICHOTOMIES
This section seeks to analyse the assessment instruments implemented in the Sinaes era, specifically in the period from 2006 (publication of the first courseassessment instrument) to 2017 (publication of the most recent instruments).
All fourteen instruments covering the regulatory acts of course authorisation, recognition and renewal of recognition were read. The analysis focused on the elements that trigger the instruments, in the following sequence: dimensions; indicators; analysis criteria; and, from 2017 onward, attributes.
Regarding the dichotomy between internal and external evaluation, although the undergraduate-course assessment instrument is essentially a document for external evaluation, elements are found that prompt reflection on, or the use of, internal evaluation to underpin the narratives and objects of assessment of indicators. This occurs more modestly in the instruments published up to 2015 and more frequently, in the attributes of the analysis criteria, in those published in 2017.
In every instrument, the evaluation commission is instructed, before the on-site visit, to examine the self-evaluation reports produced by the institution’s Internal Evaluation Committee which offers the assessed course. Moreover, the courseassessment activities include a mandatory meeting with the CPA.
From 2006 to 2015, indicators demanding self-evaluation practices as a prerequisite for external evaluation were absent or extremely incipient. There were no indicators or analysis criteria on self-evaluation in the following documents: Law Course-Assessment Instrument (Inep, 2008b), Instrument for Authorising Medicine Courses (Inep, 2010a), Instrument for Authorising Bachelor’s and Licentiate Courses (Inep, 2010b), Instrument for Authorising Law Courses (Inep, 2010c) and Instrument for Authorising Technology Courses (Inep, 2010d).
In the first published Undergraduate-Course Assessment Instrument (Inep, 2006), two indicators envisaged self-evaluation processes in their analysis criteria: “Articulation between course management and institutional management” and “Articulation between course self-evaluation and institutional self-evaluation”. In the
Instrument for Recognising Pedagogy Courses (Inep, 2010e) and the Instrument for Recognising Medicine Courses (Inep, 2010f), there is one indicator in each devoted globally to course self-evaluation. In the Course-Assessment Instrument (Bachelor’s and Licentiate) (Inep, 2008a), the Instrument for Renewing Course Recognition (Inep, 2010g), the Undergraduate-Course Assessment Instrument, On-campus and Distance (Inep, 2012) and the Undergraduate-Course Assessment Instrument - On-campus and Distance (Inep, 2015a) there is also only one indicator dealing with self-evaluation; its highest quality rating requires that the course, in exemplary fashion, has implemented academic and administrative actions arising from internal and external evaluations (including previous on-site evaluations, the Enade, the Preliminary Course Concept - CPC, etc.).
In 2017, self-evaluation features more prominently in the published instruments. Besides a specific indicator entitled “Course management and internal and external evaluation processes”, to achieve rating 5, following the additive logic of quality, numerous indicators require evidence confirming either the planned (in the case of the Assessment Instrument for the regulatory act of authorisation) or established (in the case of the Assessment Instrument for the regulatory acts of recognition and renewal of recognition) self-evaluation processes that underpin corrective and enhancement actions for future planning.
Among the indicators that demand self-evaluation to substantiate quality evidence in the assessment instruments instituted in 2017, the following stand out: “Tutoring activities”; “Knowledge, skills and attitudes required for tutoring”; “Virtual Learning Environment (AVA)”; “Performance of the course collegiate or equivalent”; “Interaction among tutors (on-site - where applicable - and distance), lecturers and distance-course coordinators”; “Student access to IT equipment”; “Basic training laboratories”; “Specific training laboratories”; “Legal practice centre: core activities and arbitration, negotiation, conciliation, mediation and real legal activities” (specific to Law courses); and “Professional environments linked to the course”.
Concerning the second dichotomy, given that one of the objectives of course evaluation is directly tied to regulation (covering the acts of authorising, recognising and renewing recognition), there is a predominance of the summative evaluation over the formative. Although the process entails drawing up a report recording, for each indicator, the justifications that substantiate the evidence gathered from document analysis, visits to facilities and meetings with various segments of the academic community (students, lecturers, tutors, managers, multidisciplinary staff, CPA, etc.), this final report is not made public by Inep; only the course’s final rating is published on the MEC website, with a stronger emphasis, therefore, on the result.
As for the third dichotomy - qualitative versus quantitative evaluation - it should be noted that in the first post-Sinaes assessment instrument, published in 2006, qualitative aspects predominate in all three dimensions aimed to undergraduate courses evaluation: didactic-pedagogical organisation; teaching, student and technical-administrative staff; and facilities.
In the documents issued between 2008 and 2015, a structural pattern emerges: qualitative, more subjective methods predominate in dimension 1 (didacticpedagogical organisation); quantitative, more objective methods prevail in dimension 2 (teaching and tutorial staff); and dimension 3 (infrastructure) strikes a greater balance between quantitative and qualitative elements. The exceptions are the 2010 authorisation and recognition instruments for Medicine courses (Inep, 2010a; Inep, 2010f), which include some quantitative indicators in dimension 1.
Taking into account the Assessment Instruments instituted between 2008 and 2015 mentioned above, in dimension 1, related to didactic-pedagogical organization, the analysis criteria of the indicators are more subjective and directed toward the nonexistence (concept 1), insufficiency (concept 2), sufficiency (concept 3), very good existence (concept 4) or excellent existence (concept 5) of proofs and evidence that justify applying the concepts to each indicator.
Quantitative elements in dimension 2 relate to greater objectivity in awarding higher ratings as the course has more master’sand doctorate-holding lecturers, a more titled and more dedicated structured teaching core (fullor part-time contracts), longer teaching and non-teaching professional experience, a higher number of fulland part-time lecturers, and more scientific, cultural, artistic or technological output. There is also more objective, quantitative assessment in indicators such as the number of subjects taught by the course’s staff, number of lecturers per place offered, number of students per class in theoretical and practical disciplines, etc.
In dimension 3, quantitative and qualitative elements intermingle. Examples of predominantly quantitative indicators include basic bibliography (relating number of copies to the course’s intake and to other courses using the same collection), journals (number subscribed and used by the course) and IT equipment (relating number of computers to the institution’s student body).
Specifically for Medicine courses, the 2010 instruments also include quantitative indicators such as the number of hospital beds available for course activities, the student-to-patient ratio in integration with the Unified Health System (SUS) and the number of accredited health-area courses at the institution with an Enade rating of at least 4.
It is worth noting that in the instruments published from 2006 to 2015, qualitative aspects were also to be recorded in the overall narrative for a set of indicators or for each dimension as a whole.
From 2017 onward, the logic of the assessment instruments is organised almost entirely around qualitative indicators in all three dimensions: didactic-pedagogical organisation; teaching and tutorial staff; and infrastructure. To assign ratings to each indicator, a set of required attributes is established within the analysis criteria, as already mentioned. Thus, the more attributes a course possesses, the higher its rating for the indicator. The exceptions are three quantitatively analysed indicators in dimension 2: the Structured Teaching Core - NDE (whose rating rises as lecturers are more titled and more dedicated in workload terms), tutor qualification and training (where higher ratings correspond to a degree in the course’s field and higher academic qualifications), and scientific, cultural, artistic or technological output (where the rating increases with the number of publications by each lecturer in the previous three years). The 2017 instruments also include a field for evaluators to provide a brief qualitative analysis of each dimension.
Regarding the fourth dichotomy, integral or point-in-time evaluation, since the instruments analysed pertain to one of the three Sinaes pillars, the UndergraduateCourse Assessment (ACG), a preliminary analysis shows that, in this specific respect, the evaluation is point-in-time, as it does not cover the entire system. This is corroborated by the fact that the assessment instruments for undergraduate courses lack, in some periods, or contain very few, indicators linking course evaluation to the other Sinaes pillars. Up to 2017, such correlation, when present, was extremely limited or incipient, relating only to self-evaluation and the Enade, thereby establishing little dialogue between the course-assessment instruments and the other pillars of Sinaes. From 2017 onward, there is a trend in the instruments toward greater correlation between undergraduate-course evaluation and the other pillars, chiefly self-evaluation, which is incorporated not only in a specific indicator but also as a key attribute for rating 5 in several indicators highlighted earlier in the discussion of internal versus external evaluation.
Lastly, another element that characterises point-in-time evaluation is the near absence of correlation between the assessment instruments and the Ministry of Education’s information tools. Moreover, in terms of making results public, the logic again resembles point-in-time evaluation, as disclosure is framed as an established truth based on the final rating awarded.
5 FINAL CONSIDERATIONS
Over its twenty-year existence, the National Higher Education Evaluation System (Sinaes) can be said to have brought important advances to the improvement of higher education in Brazil as a public policy, establishing an evaluation framework structured around three central pillars: institutional evaluation (comprising external evaluation and self-evaluation), undergraduate-course evaluation and the national student performance examination.
Relating specifically to the second pillar, undergraduate-course evaluation, it became closely aligned with regulatory processes required for the authorisation, recognition and renewal of recognition of courses.
Based on the analysis of the undergraduate-course assessment instruments implemented within Sinaes between 2006 and 2017, fourteen instruments were identified, showing greater or lesser incidence of evidence in the spheres studied, namely the evaluation dichotomies: internal/external, formative/summative, qualitative/quantitative and integral/point-in-time.
The integration between self-evaluation and external evaluation is crucial for fostering a broader, contextualised view of the quality of undergraduate courses offered nationwide. In parallel, the use of more qualitative indicators and attributes from 2017 onward demonstrates a shift towards a less data-driven and more holistic evaluation, which, in turn, leans more on subjectivity and the evaluators’ interpretative logic in assigning ratings.
Thus, Sinaes in general, and the assessment instruments used for undergraduate-course evaluation in particular, play a fundamental role not only in regulation but also in the continuous development and enhancement of educational quality. Nonetheless, it is essential that Inep continue refining these instruments, promoting greater articulation among the different aspects of evaluation to achieve an integrated and effective view of higher education in the country.
-
Translated by: Isabel Codá de Simas isabel.coda@gmail.com
Data availability
The data underlying this research are contained within the article.
REFERENCES
- BELLONI, Isaura; MAGALHÃES, Heitor de; SOUSA, Luzia Costa de. Metodologia de avaliação em políticas públicas: uma experiência em educação profissional. São Paulo: Cortez, 2001.
-
BRASIL. Comissão Especial de Avaliação. SINAES - Sistema Nacional de Avaliação da Educação Superior: bases para uma nova proposta de avaliação da educação superior. Brasília: 2004a. Disponível em: https://download.inep.gov.br/publicacoes/institucionais/avaliacoes_e_exames_da_educacao_superior/bases_para_uma_nova_proposta_de_avaliacao_da_educacao_superior_brasileira.pdf Acesso em: 10 out. 2024.
» https://download.inep.gov.br/publicacoes/institucionais/avaliacoes_e_exames_da_educacao_superior/bases_para_uma_nova_proposta_de_avaliacao_da_educacao_superior_brasileira.pdf -
BRASIL. Ministério da Educação. Lei nº 10.861, de 14 de abril de 2004. Institui o Sistema Nacional de Avaliação da Educação Superior - SINAES e dá outras providências. Diário Oficial da União, Brasília, DF, 15 abr. 2004b. Disponível em: https://www.planalto.gov.br/ccivil_03/_ato2004-2006/2004/lei/l10.861.htm Acesso em: 06 set. 2024.
» https://www.planalto.gov.br/ccivil_03/_ato2004-2006/2004/lei/l10.861.htm -
BRASIL. Secretaria-Geral. Lei nº 14.375, de 21 de junho de 2022. Altera as Leis nºs 10.260, de 12 de julho de 2001, 10.522, de 19 de julho de 2002, e 12.087, de 11 de novembro de 2009, para estabelecer os requisitos e as condições para realização das transações resolutivas de litígio relativas à cobrança de créditos do Fundo de Financiamento Estudantil (Fies), a Lei nº 10.861, de 14 de abril de 2004, para estabelecer a possibilidade de avaliação in loco na modalidade virtual das instituições de ensino superior e de seus cursos de graduação, a Lei nº 13.988, de 14 de abril de 2020, para aperfeiçoar os mecanismos de transação de dívidas, e a Lei nº 13.496, de 24 de outubro de 2017; e revoga dispositivos das Leis nºs 13.530, de 7 de dezembro de 2017, 13.682, de 19 de junho de 2018, 13.874, de 20 de setembro de 2019, e 14.024, de 9 de julho de 2020. Diário Oficial da União, Brasília, DF, 21 jun. 2022. Disponível em: https://www.planalto.gov.br/ccivil_03/_ato20192022/2022/lei/l14375.htm Acesso em: 06 set. 2024.
» https://www.planalto.gov.br/ccivil_03/_ato20192022/2022/lei/l14375.htm - DIAS SOBRINHO, José. Avaliação institucional: integração e ação integradora. Avaliação: Revista de Avaliação da Educação Superior, Campinas; Sorocaba, v. 2, n. 2, p. 19-29, jun. 1997. Disponível em: v02n02a03.pdf Acesso em: 10 out. 2024.
- DIAS SOBRINHO, José. Avaliação da educação superior Petrópolis (RJ): Vozes, 2000.
- DIAS SOBRINHO, José. Avaliação da educação superior, regulação e emancipação. In: DIAS SOBRINHO, José, RISTOFF, Dilvo (orgs). Avaliação e compromisso público: a educação superior em debate. Florianópolis (SC): Insular, 2003. p. 35-52.
-
INEP. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. Instrumento de Avaliação de Cursos de Graduação. Brasília: Inep, 2006. Disponível em: https://download.inep.gov.br/publicacoes/institucionais/avaliacoes_e_exames_da_educacao_superior/instrumento_de_avaliacao_de_cursos_de_graducao.pdf Acesso em: 10 out. 2024.
» https://download.inep.gov.br/publicacoes/institucionais/avaliacoes_e_exames_da_educacao_superior/instrumento_de_avaliacao_de_cursos_de_graducao.pdf -
INEP. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. Instrumento de Avaliação de Cursos de Graduação (Bacharelado e Licenciatura). Brasília: Inep, 2008a. Disponível em: https://download.inep.gov.br/download/superior/condicoesdeensino/2010/instrumento_reconhecimento_bacharelado_licenciatura3.pdf Acesso em: 10 out. 2024.
» https://download.inep.gov.br/download/superior/condicoesdeensino/2010/instrumento_reconhecimento_bacharelado_licenciatura3.pdf -
INEP. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. Instrumento de Avaliação do curso de Direito: subsidia o ato de reconhecimento. Brasília: Inep, 2008b. Disponível em: https://download.inep.gov.br/download/superior/condicoesdeensino/2010/instrumento_reconhecimento_curso_direito2.pdf Acesso em: 10 out. 2024.
» https://download.inep.gov.br/download/superior/condicoesdeensino/2010/instrumento_reconhecimento_curso_direito2.pdf -
INEP. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. Instrumento de avaliação do curso de Medicina: subsidia o ato de autorização. Brasília: Inep, 2010a. Disponível em: https://download.inep.gov.br/download/superior/condicoesdeensino/2010/instrumento_autorizacao_curso_medicina2.pdf Acesso em: 10 out. 2024.
» https://download.inep.gov.br/download/superior/condicoesdeensino/2010/instrumento_autorizacao_curso_medicina2.pdf -
INEP. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. Instrumento para Autorização de Bacharelado e Licenciatura. Brasília: Inep, 2010b. Disponível em: https://download.inep.gov.br/download/superior/condicoesdeensino/2010/instrumento_autorizacao_bacharelado_licenciatura2.pdf Acesso em: 10 out. 2024.
» https://download.inep.gov.br/download/superior/condicoesdeensino/2010/instrumento_autorizacao_bacharelado_licenciatura2.pdf -
INEP. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. Instrumento para Autorização de Curso de Direito. Brasília: Inep, 2010c. Disponível em: https://download.inep.gov.br/download/superior/condicoesdeensino/2010/instrumento_autorizacao_curso_direito2.pdf Acesso em: 10 out. 2024.
» https://download.inep.gov.br/download/superior/condicoesdeensino/2010/instrumento_autorizacao_curso_direito2.pdf -
INEP. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. Instrumento para Autorização de Curso Tecnológico Brasília: Inep, 2010d. Disponível em: https://download.inep.gov.br/download/superior/condicoesdeensino/2010/instrumento_autorizacao_curso_tecnologico2.pdf Acesso em: 10 out. 2024.
» https://download.inep.gov.br/download/superior/condicoesdeensino/2010/instrumento_autorizacao_curso_tecnologico2.pdf -
INEP. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. Instrumento para Reconhecimento de Curso de Pedagogia Brasília: Inep, 2010e. Disponível em: https://download.inep.gov.br/download/superior/condicoesdeensino/2010/instrumento_reconhecimento_curso_pedagogia2.pdf Acesso em: 10 out. 2024.
» https://download.inep.gov.br/download/superior/condicoesdeensino/2010/instrumento_reconhecimento_curso_pedagogia2.pdf -
INEP. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. Instrumento para Reconhecimento de Curso de Medicina Brasília: Inep, 2010f. Disponível em: https://download.inep.gov.br/download/superior/condicoesdeensino/2010/instrumento_reconhecimento_curso_medicina2.pdf Acesso em: 10 out. 2024.
» https://download.inep.gov.br/download/superior/condicoesdeensino/2010/instrumento_reconhecimento_curso_medicina2.pdf -
INEP. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. Instrumento para Renovação de Reconhecimento de Cursos Brasília: Inep, 2010g. Disponível em: https://download.inep.gov.br/download/superior/condicoesdeensino/2010/instrumento_renovacao_reconhecimento_cursos2.pdf Acesso em: 10 out. 2024.
» https://download.inep.gov.br/download/superior/condicoesdeensino/2010/instrumento_renovacao_reconhecimento_cursos2.pdf -
INEP. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. Instrumento de Avaliação de Cursos de Graduação: presencial e a distância. Brasília: Inep, 2012. Disponível em: https://download.inep.gov.br/educacao_superior/avaliacao_cursos_graduacao/instrumentos/2012/instrumento_com_alteracoes_maio_12.pdf Acesso em: 10 out. 2024.
» https://download.inep.gov.br/educacao_superior/avaliacao_cursos_graduacao/instrumentos/2012/instrumento_com_alteracoes_maio_12.pdf -
INEP. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. Instrumento de Avaliação de Cursos de Graduação - presencial e a distância. Brasília: Inep, 2015a. Disponível em: https://download.inep.gov.br/educacao_superior/avaliacao_cursos_graduacao/instrumentos/2015/instrumento_cursos_graduacao_publicacao_agosto_2015.pdf Acesso em: 10 out. 2024.
» https://download.inep.gov.br/educacao_superior/avaliacao_cursos_graduacao/instrumentos/2015/instrumento_cursos_graduacao_publicacao_agosto_2015.pdf -
INEP. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. Sistema Nacional de Avaliação da Educação Superior (Sinaes). Brasília: Inep, 2015b. v. 5. Disponível em: https://download.inep.gov.br/publicacoes/institucionais/avaliacoes_e_exames_da_educacao_superior/sinaes_volume_5_avaliacao_in_loco_referenciais_no_ambito_do_sinaes.pdf . Acesso em: 20 set. 2024.
» https://download.inep.gov.br/publicacoes/institucionais/avaliacoes_e_exames_da_educacao_superior/sinaes_volume_5_avaliacao_in_loco_referenciais_no_ambito_do_sinaes.pdf -
INEP. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. Instrumento de Avaliação de cursos de graduação - Presencial e a distância - Autorização Brasília: Inep, 2017a. Disponível em: https://download.inep.gov.br/educacao_superior/avaliacao_cursos_graduacao/instrumentos/2017/curso_autorizacao.pdf Acesso em: 20 set. 2024.
» https://download.inep.gov.br/educacao_superior/avaliacao_cursos_graduacao/instrumentos/2017/curso_autorizacao.pdf -
INEP. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. Instrumento de Avaliação de cursos de graduação - Presencial e a distância - Reconhecimento Renovação de Reconhecimento Brasília: Inep, 2017b. Disponível em: https://download.inep.gov.br/educacao_superior/avaliacao_cursos_graduacao/instrumentos/2017/curso_reconhecimento.pdf Acesso em: 20 set. 2024.
» https://download.inep.gov.br/educacao_superior/avaliacao_cursos_graduacao/instrumentos/2017/curso_reconhecimento.pdf -
INEP. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. O instrumento de avaliação: conceituação, seções, lógica e uso. Brasília: Inep, 2018. Disponível em: https://files.cercomp.ufg.br/weby/up/65/o/03_A_l%C3%B3gica_do_Instrumento_de_Avalia%C3%A7%C3%A3o-Autoriza%C3%A7%C3%A3o-12_03.pdf Acesso em: 10 out. 2024.
» https://files.cercomp.ufg.br/weby/up/65/o/03_A_l%C3%B3gica_do_Instrumento_de_Avalia%C3%A7%C3%A3o-Autoriza%C3%A7%C3%A3o-12_03.pdf - LEITE, Denise. Reformas universitárias: avaliação institucional participativa. Petrópolis (RJ): Vozes, 2005.
- RISTOFF, Dilvo. Revisitando o SINAES: Sistema Nacional de Avaliação da Educação Superior. Florianópolis: Insular, 2024.
- RISTOFF, Dilvo; GIOLO, Jaime. O Sinaes como Sistema. Revista Brasileira de PósGraduação (RBPG), Brasília, v. 3, n. 6, 2006.
-
Editor de seção:
André Pires
-
Editor de Layout:
Silmara Pereira da Silva Martins
