Adaptation and evidences of validity of the Remote Teaching Reactions Scale in Exact SciencesAbstract
The objective of this study was to adapt and verify the validity evidence of a Reactions Scale to Instructional Procedures in Distance Education. The investigation was carried out in a Brazilian federal public university and 507 students participated in the study. Descriptive and exploratory factor analyses were used. The Scale of Reactions to Instructional Procedures in Remote Teaching was obtained, with a KMO of 0.91 and two factors. Factor 1, Instruction Planning, grouped 10 items, with an excellent internal consistency index of 0.90 and factor loadings between 0.43 and 0.75. Factor 2, called Instructional Events showed an internal consistency index of 0.82 for 2 items, with factor loadings of 0.81 and 0.82. The suggestion to maintain the bifactorial structure was motivated based on Gagné’s theory, which can support the proposition of new measures in the Evaluation of Educational Actions at the Reaction level.
Keywords
Education, distance; Educational measurement; Engineering; Evidence synthesis
Resumo
O objetivo deste estudo é adaptar e verificar evidências de validade de uma escala de Reações aos Procedimentos Instrucionais em Educação a Distância. A pesquisa foi realizada em uma universidade pública federal brasileira com a participação de 507 estudantes. Foram utilizadas análises descritivas e fatoriais exploratórias. Obteve-se a Escala de Reações aos Procedimentos Instrucionais em Ensino Remoto, com índice KMO de 0,91 e dois fatores. O Fator 1, Planejamento da Instrução, agrupou 10 itens, com excelente índice de consistência interna de 0,90 e cargas fatoriais entre 0,43 e 0,75. O Fator 2, denominado Eventos da Instrução, apresentou um índice de consistência interna de 0,82 para 2 itens, com cargas fatoriais 0,81 e 0,82. A proposta de manter a estrutura bifatorial foi motivada pela teoria de Gagné que pode colaborar com a proposição de novas medidas em Avaliação de Ações Educacionais no nível de Reação.
Palavras-chave
Educação a distância; Avaliação educacional; Engenharia; Síntese de evidência
The use of digital technologies in the offering of professional trainings, undergraduate courses and business training has enabled new teaching and learning experiences in public and private institutions. The adoption of e-learning, a teaching method mediated by the internet tools, has grown with the aim of reaching a large number of individuals, geographically dispersed and with different routines (Sun & Chen, 2016). Online learning platforms, social networks, interpersonal communication applications such as WhatsApp and educational software are examples of digital technologies, as understood in this article (Dolch & Zawacki-Richter, 2018).
In view of the global health scenario during and after the COVID-19 pandemic and the abrupt changes that higher education has undergone, the debate on teaching modalities and the use of digital technologies in the teaching-learning process has been intensified and is likely to continue for a long time (Watermeyer et al., 2021; Zhang et al., 2020). Educational institutions that chose to make this change did so in an improvised and compulsory manner, in order not to disrupt the progress of teaching activities. The offering of curricular subjects in this model, was called, in the national and international literature, Emergency Remote Teaching (ERT) (Amaral & Polydoro, 2020; Murphy et al., 2020; Rapanta et al. 2020). Most educational institutions in the world implemented it without instructional planning, according to the infrastructure conditions and faculty and student body available (Nguyen et al., 2020; Peimani & Kamalipour, 2021).
Considering today’s educational context and the number of variables involved in the provision of higher education during and after the pandemic, evaluating the phenomena of remote teaching, not just emergency teaching, helps in the process of continuous evaluation, at a political and pedagogical level, of the educational actions of Higher Education Institutions (HEI) (Ferreira & Mourão, 2020).
Based on these academic and practical motivations, the objective of this article was to adapt and verify evidence of validity of the Scale of Reactions to Instructional Procedures in Distance Education by Martins and Zerbini (2015). The main objective was to support decision-makers, in the field of Exact Sciences teaching, regarding the formatting and offering of courses in remote mode that deliver effective results.
Our article will focus on the Calculus 2 discipline, offered to a large number of undergraduate students in Exact Sciences, mainly in Engineering. However, it is worth noting that, although the scope of this study was limited to this discipline, the scale proposed in this investigation can be replicated in other disciplines, in different areas of knowledge, at any level or modalities of education. The scale will be called “Reactions to Instructional Procedures in Remote Education”.
The relevance lies, first of all, in the fact that this research seeks to contribute to improving the quality of online educational activities in undergraduate Engineering courses. In addition, it proposes to promote an intersection between different areas of knowledge: Training, Development and Education (TD&E) and Production Engineering/Engineering Education.
The provision of distance learning undergraduate courses in Engineering in federal public HEI is a subject that in the current practice is still under construction; thus, scientific production on the topic has been progressing at a reduced pace, resulting in few Brazilian studies addressing online learning in these institutions (Assumpção et al., 2018). The fact that there are less Brazilian research papers available may be due to the fact that, before the pandemic, many foreign universities already offered fully online or hybrid courses for training engineers.
Abreu and Oliveira-Melo (2020) reported some Brazilian studies on Calculus online tutoring before the pandemic. According to the authors, this practice has already been adopted in Engineering courses at some public HEI. However, many of the results presented by the authors showed that, before the pandemic, students’ demand for learning via virtual environments was meagre. We can observe that both Brazilian and foreign researchers have shown strong interest in topics related to the incorporation of digital technologies in Engineering teaching.
According to Meneses et al. (2010), courses mediated by digital technologies tend to be based on instructional and cognitive psychology to compose the curricular structure, as they facilitate the description of competencies and the understanding of how the teacher can enable the internal learning processes of individuals in the framework of online teaching. Given that there are many theories produced and widely used, this article will briefly describe two instructional theories that are closely related to the object of this investigation and have a cognitivist basis. The first of them is that of Bloom et al. (1972) and the second is that of Gagné (1985).
In order to understand the human learning process and, based on this awareness, understand how it can be measured, Bloom et al. (1972) classify learning outcomes into three domains: 1) Cognitive domain, which encompasses a set of intellectual activities ordered according to their levels of complexity; 2) Affective domain, which includes attitudes, values, interests and emotional tendencies that guide human behavior in social relationships, and its level of achievement is measured by the degree of internalization or incorporation; 3) Psychomotor domain, which indicates the degree of automation of certain motor or muscular activities. For each domain, that is, for each expected learning outcomes, appropriate instructional means, strategies and procedures should be selected. And each domain has also its own classifications (or “taxonomies”, as Bloom et al. (1972) put it). Depending on the type of course or discipline, one domain prevails over the other and each requires specific situations favorable to learning.
Thus, Bloom’s Taxonomies suggest that there is a hierarchical and cumulative relationship between the categories that make up each domain. In the case of the cognitive domain, for example – widely explored in Calculus and Mathematics – the expected learning outcomes, according to the theory, are linked to the intellectual tasks – from the ability to memorize a concept (Level 1) to the ability to make judgments and criticize the concept (Level 6). The levels of the cognitive domain are: Level 1: Knowledge; Level 2: Comprehension; Level 3: Application; Level 4: Analysis; Level 5: Synthesis; and Level 6: Evaluation.
The second instructional theory adopted as a basis for our investigation is that of Gagné (1985) – one of the foremost exponents of the cognitivist line. He mentions two concepts: internal events and external events for learning. According to this author, with each course or discipline offered, the teacher must provide conditions to facilitate the student’s internal learning process. To do so, the teacher must pay attention to the following instructional events: 1) Obtain the learner’s attention; 2) Inform the educational objective; 3) Remind the learner of prerequisites; 4) Present stimulus to the learner; 5) Provide learning guides; 6) Provoke performance; 7) Provide feedback; 8) Evaluate performance; 9) Increase retention and transfer.
According to Gagné (1985), internal events relate to the student’s degree of readiness to learn and this can vary greatly when considering the personal characteristics of individuals or the way in which the student is induced to learn. Thus, instruction (course or subject, for example) must be planned taking into account that these internal learning processes go hand in hand with the conditions provided by the teacher. Thus, for each domain and level of result proposed by Bloom, the appropriate instructional procedures and means must be selected to execute all the instructional events mentioned by Gagné (1985).
Current practice has shown that many online courses have been offered without systematic instructional planning. This is a cause for concern, since in the case of distance learning, high investments are made in training of this nature. Furthermore, ineffective planning of any educational activity can compromise learning outcomes or their measurement.
Regarding the process of evaluating the planning, efficacy and effectiveness of TD&E actions, production of knowledge began in the 1970s with the now traditional training evaluation models developed by Kirkpatrick (1976) and Hamblin (1978). From these models, other important models emerged that are now widely used in national and international literature: Borges-Andrade (2006), Noe and Schmitt (1986), Phillips and Gully (1997), Abbad (1999).
In the world literature, among the models that encompass variables that predict results, two stand out: the CIRO Model – Context, Input, Reaction and Outcome by Warr et al. (1970); the ISAM Model – Integrated and Summative Assessment Model by Borges-Andrade (2006). The ISAM Model was chosen as a reference for this thesis because: 1) It originated in the educational context; 2) It was developed by a Brazilian author and was adapted and applied to the national context; 3) It is based on Instructional Psychology, which also supports this work; 4) It encompasses the Procedures component (teacher planning strategies) explicitly and at the same time interconnected with the Processes component (student behavior during the course). It is explicitly mentioned because variables on teaching procedures are not clearly present in the other models; 5) It is integrated (points to feedback) and formative (raises theoretical discussions and guides practice).
The operational definitions of the assessment components proposed in the ISAM Model have been and continue to be tested by national and foreign researchers in different contexts – corporate, professional qualification and teaching, such as these most recent studies: Alsabahi et al. (2021) – corporate training in the private sector; Carlisle et al. (2019) – training in the public sector; Nauman et al. (2020) – corporate training in the private sector; Sade et al. (2020) – continuing education in the nursing field.
Reaction measures, which are the focus of our study, according to national and international literature, refer to students’ opinions or satisfaction with the different aspects of the educational activity in which they participated. Thus, student satisfaction can be analyzed in relation to the: teacher’s instructional procedures, graphical interface of a distance learning course or tutor’s performance. Reaction is the first level of evaluation of a subject or course, considering the five levels proposed by traditional models. Mayer (2019) states that some determining factors of student satisfaction with face-to-face education seem to also apply to online education, such as motivation to learn, quantity and quality of content, and support. A significant amount of research before the pandemic on the success factors in online education is available.
In the national literature there are reports of researchers in the areas of TD&E and psychology who have developed psychometric instruments to measure individuals’ reactions to different dimensions associated with the characteristics of in-person and distance learning education. Many of these scales were developed and statistically validated for the first time between 1998 and 2008: Abbad et al. (2000) – Reaction to the Course; Carvalho and Abbad (2006) – Reaction to Tutor Performance, Reaction to the Graphical Interface, Reaction to Results and Applicability; Zerbini and Abbad (2009) – Reaction to Instructional Procedures and Reaction to Tutor Performance; Zerbini and Abbad (2008) – Learning Strategies. The evidence of validity of these scales has been investigated in different teaching contexts.
In the international literature, some studies have evaluated student reactions to online higher education learning experiences and found significantly positive correlations between student satisfaction and other variables such as course program and interaction with peers (Alqurashi, 2019). What was found in the analysis of the empirical literature on evaluation at the Reaction level in online courses was that, mainly in studies in the engineering context, studies that evaluate student reaction to the graphical interface or to the course predominate, with analysis of variables such as: quality of the teaching platform, course design, and perceived usefulness of the content (Eom & Ashill, 2018).
Hence, in this article reaction to instructional procedures in remote teaching is considered a measure of the participants’ satisfaction regarding instructional characteristics of the course such as: quality of teaching objectives, content, sequence, learning assessments, as well as teaching strategies and means and the use of web tools by the teacher (Martins et al., 2018).
Method
The study was registered on the Plataforma Brasil and approved by the Research Ethics Committee of Universidade Federal de Itajubá (UNIFEI, Federal University of Itajubá, Brazil) (CAAE/Opinion No. 4,573,531).
Study Context and Profile of the Target Institution
The UNIFEI is a century-old higher education institution where students have been graduating in engineering since 1913. It became a Federal institution in 1956 and the project to transform it into a university was implemented on April 24, 2002, under Law nº 10.435 issued by the President of the Republic, Fernando Henrique Cardoso. Over time, UNIFEI has expanded its offerings of several undergraduate engineering specialties as well as courses in Exact Sciences, such as Physics, Mathematics and Chemistry.
In 2008, a pioneering partnership between the local government (Itabira municipal government), the private sector (Vale do Rio Doce), the Ministry of Education and UNIFEI made it possible to set up a UNIFEI campus in Itabira, Minas Gerais. Today, this campus offers nine undergraduate courses in Engineering. The main campus offers 25 on-site undergraduate courses, one distance learning course – a Bachelor’s Degree in Physics – since 2007, and 23 lato sensu and stricto sensu graduate courses. There are specializations in four areas, offered entirely online through the federal government’s program: Universidade Aberta do Brasil.
Features of Calculus 2
In a survey carried out by UNIFEI’s in February 2021, it was found that, on the Itajubá campus, the pedagogical projects of the bachelor’s degree courses in Mathematics, Chemistry, Physics, Computer Science, Atmospheric Sciences and 13 of the 14 Engineering courses include Calculus 2 (MAT002) as a mandatory credit component. In total numbers, this means 18 of the 26 undergraduate courses offered on the main campus now require MAT002 prior credit. Therefore, the chosen subject for our investigation was Calculus 2.
Population and Sample
The population of this study was composed of 1,007 individuals, which represent the number of students who took Calculus 2 at UNIFEI in Itajubá in the second semester of 2020 in the ERT modality. The sample included 507 students. The sample calculation was made based on the assumptions required by the statistical analysis techniques that were intended to be adopted. For example, because factor analyses would be performed, the criterion suggested by Pasquali (2009) was considered i.e. 10 cases for each observable variable. The number of participants involved covered 51% of the target population.
The criteria for selecting the study population and the sample, as well as the participants’ forms of access were as follows: 1) The sampling procedure was non-probabilistic, with individual data collection, using a single electronic questionnaire formulated through Google Forms (containing the scale questions and questions to collect information about the sociodemographic profile of the respondents); 2) The population included all students who took Calculus 2 in the second semester of 2020 (N = 1,042); 3) The participants were informed, in the questionnaire header, about the objectives of the study and about the anonymity and confidentiality of their responses; 4) Google Forms resources were used to ensure that the respondents were required to sign a Free and Informed Consent Form, as well as the requirement that answering all the questions was mandatory, thus avoiding omissions. On average, it took 12 minutes to answer the questionnaire.
Over 80% of the students were between 18 and 24 years old; a significant number of students were still studying Calculus 2 in the 7th semester of their undergraduate course; a total 87% of the students lived in their family homes. It can be observed that some variables indicate a certain homogeneity among the participants’ profiles, such as the fact that 97% of them did not have any children, 83% were between 18 and 24 years old, 87% returned to their family home during the pandemic, and 74% did not receive material or financial support from the institution.
Research Instruments
The measurement instruments used in this investigation were: 1) A questionnaire that sought to review the academic and socioeconomic profile of students; 2) The Reaction to Instructional Procedures in Remote Teaching scale, adapted from Martins and Zerbini (2015).
Borsa et al. (2012) consider that there are many advantages in the process of adapting an existing scale, instead of developing a new scale for a specific context. By opting for adaptation, the investigator is able to compare the results obtained in different samples and contexts, which allows for an impartial and fair assessment, and a greater capacity for generalization, given that it is the same measure, which evaluates the same construct, based on a single theoretical and methodological perspective.
The task of semantic and theoretical validation of the items was assigned to two judges: a Doctorate’s student in Psychology with a major in TD&E from the University of São Paulo (Ribeirão Preto), and a professor-researcher from University. The scale originally used an 11-point Likert-type scale format, ranging from 0 to 10. This was the first change made. The seven-point scale, which has been most suggested in the foreign literature in recent years, was adopted here (Simms et al., 2019).
The first highlight concerns the inclusion of the item “Variety in the use of innovative educational resources, such as live classes, videos made by the teacher, use of software and applications”. In this regard, the intention was to observe whether these digital technologies were used during the semester. Given the exclusively online format of the courses offered, the use of these resources would be predictable and, from an instructional point of view, desirable, although it is not mandatory to conduct a distance course. The hypothesis was considered that, at the time, many teachers migrated to another modality while maintaining the same traditional procedures adopted before the pandemic, as reported by Naji et al. (2020). In other words, those teachers may have not explored these tools. Thus, the aim was to identify whether, in the framework of our study, those tools were used.
The items “News and reminders posted in the virtual learning environment (VLE)” and “Links made available in the Virtual Learning Environment (VLE)” were removed. Regarding the second item, initially, it was considered that, although UNIFEI has an academic system (SIGAA), it was not known whether the Calculus 2 teachers used it as a VLE, as occurs in traditional distance learning systems. In other words, the VLE in distance learning is not only a software for forwarding materials and for tasks requests; through its resources, it is possible to interact asynchronously and synchronously with students, monitor the entire learning process and, in some systems, generate reports on student performance and progress. Tools that allow the creation of chats (chat rooms), polls and questionnaires are also common on these platforms and are available in UNIFEI’s SIGAA. Software resources for automatic correction of questions’ answers are also present in this system. However, according to verbal information provided by one of the Calculus 2 teachers, most of the teachers in this area used the platform only to forward study materials or tests/exams. Some teachers adopted Moodle, for the same purposes and the hosting of such materials was done through file uploads and not through links. Finally, given that the scale had the item “Readings recommended by the teacher” and, added to this information from the teacher, it was decided to remove the item related to links.
The items “News forums” and “Discussion forums”, which were part of the original scale, were excluded. These resources are available in the academic system. However, although the UNIFEI Administration determined that, during the pandemic, SIGAA should be used as the official system, the possibility that Calculus 2 teachers would not use these features led to the decision to remove these items so as not to “inflate” the questionnaire.
Finally, the final version of the instrument of this research “Reaction to instructional procedures in remote teaching” had 12 items, associated with a seven-point Likert-type scale, ranging from 1 (Very bad) to 7 (Very good). The aim was to evaluate aspects related to instructional planning through measures related to the procedures adopted by the teacher and to some phenomena linked to the nature of the subject evaluated.
Data Collection Procedures
The survey questionnaire was distributed to undergraduate students on different occasions: (a) first contact through a direct mailing (December 2020); (b) replication of this campaign by the UNIFEI Communications department in the official email “UNIFEI Disclosure”, which was forwarded to all undergraduate students on the Itajubá campus (December 2020 and January 2021); (c) a direct mail to scholarship holders through the Student Affairs Office (February 2021); (d) distribution of the questionnaire by six professors who taught technical subjects in the second semester of 2020 to the same students who were the target audience of the study. In this case, these professors voluntarily made the survey link available in their virtual classes on SIGAA (February 2021).
Finally, the last campaign was the result of a partnership with the UNIFEI Information Technology Department (ITD), which, with the consent of the Pro-Dean of Undergraduate Studies, hosted the questionnaire link on SIGAA in March 2021. Upon opening the system, undergraduate students were invited to answer the questionnaire at that time or, later, when they logged in again. In this case, the ITD ensured that it only allowed one more notification about the questionnaire, so as not to harm the student’s experience with the system interface.
All of these data collection procedures were carried out remotely. With each release, the texts accompanying the calls were adapted to the media in order to attract the largest possible number of respondents. Attention was paid to the language used and to forwarding the surveys at strategic times and periods. In addition, there was concern about the objectivity in the preparation of the messages, as well as the interface of the questionnaire, such as adapting the layouts of the texts and questions according to each type of media: for smartphones and for computers.
Data Analysis Procedures
Initially, it should be clarified that it was not necessary to perform treatment of missing cases because the questionnaire format eliminated the possibility of empty cells in the database. Thus, the first step in this phase was to analyze, through descriptive statistics, simple parameterization (mean, mode, standard deviation, etc.). Then, the responses to these 12 variables were standardized in Z scores. In this research, it was chosen to standardize the data in order to work with eigenvalues and eigenvectors of the correlation matrix (R), instead of the covariance matrix.
According to Tabachnick and Fidell (2013), the transformation into a Z score is essential to identify univariate and multivariate outliers (extreme cases). To detect them, the following criterion was considered: responses whose standardized scores were equal to or greater than 3.29 (p < 0.001, two-tailed) would be removed. This means that the elimination of any type of outlier implied the exclusion of individuals from the database.
Therefore, after cleaning the database, the technique for reducing the dimensionality of the data matrix was chosen. However, the Principal Component Analysis (PCA) was also performed in an exploratory manner to identify the number of possible components in advance. After the PCA, with information on the number of components that could be retained in the data matrix of each psychometric scale, the factor structures were confirmed using the Exploratory Factor Analysis (EFA). At this stage, three steps were completed: 1) Decision on the method for estimating factor loadings; 2) Definition of the rotation procedure; 3) Calculation of the factor scores. The Principal Axis Factoring (PAF) method was used.
The next stage of the EFA was the selection of the type of factor rotation. The oblique rotation method (direct oblimin) was adopted, which better suits the nature of this study, when compared to orthogonal rotation methods such as varimax and equimax. In all the PCA and EFA factor analyses, the software IBM®SPSS®, version 26, was used, because it allows the application of this rotation procedure.
Finally, to define the number of factors, it is also necessary to judge their relevance. This judgment is made after the rotation procedure (Tabachnick & Fidell, 2013). Based on the guidance of these authors, three relevance criteria were adopted: theoretical relevance of the factor, individual variance, and internal consistency index. Theoretical relevance was assessed by interpreting the relationship between the factor and the construct; the factors obtained in previous studies were also compared. In this part, the collaboration of the scale author was essential to understand the factor structures that were formed.
Regarding the individual variance, the Harman criterion was adopted, in which the factor that explains at least 3% of the total variance explained by the variables is considered relevant (Tabachnick & Fidell, 2013). The size of the factor loadings was also verified because it expresses the stability of the factor, that is, the larger the loadings are, the more the variable is representative of the factor. However, a factor may be valid but not consistent, so internal consistency analyses of the factors were also performed by calculating Cronbach’s alpha index.
After the EFA, the stability and interpretability of the scale items were verified. Stability refers to the analysis of how well the items represent the factor; such analysis generally takes into account the factor loading of the item (or independent variable). Only items with factor loadings greater than or equal to 0.40 were maintained. The items interpretability and the factors to which they were linked were assessed by reviewing the literature in this area.
Results
The scale “Reaction to instructional procedures in remote teaching” contains 12 items and was evaluated on a 7-point Likert-type scale that measured the quality criteria (1 – Very poor to 7 – Very good). The responses to the scale items did not present univariate outliers; eight cases of multivariate outliers were identified and removed from the data file; the presence of linear relationships between variables was identified; there was no case of multicollinearity. The PCA suggested an empirical structure with two components that together explained 60% of the total variance in the participants’ responses to the questionnaire items. Subsequently, the behavior of the components was also analyzed through Parallel Analysis. The results confirmed the possibility of extracting up to two components.
The EFA was then performed with extraction by main axes. Two possibilities were tested: the unifactorial and the bifactorial structure. However, given the context of the study and the perspective of raising a discussion in the theoretical and methodological field of Instructional Psychology and TD&E, the bifactorial structure proved to be more appropriate, according to data in Table 1.
The scale presented a KMO of 0.91, considered excellent, and two factors. Factor 1, called Instructional Planning that grouped 10 items, with an excellent internal consistency index (α = 0.90) and factor loadings between 0.43 and 0.75. These items measure traditional procedures that are adopted when planning a course or discipline, for example, selecting a teaching modality, content sequence, objectives, proposed activities, etc. It is worth mentioning that, regarding items 3 – “Sequence of presentation of the discipline content” and 11 – “Discipline program”, it is highlighted that teachers or those responsible for preparing the discipline program are mostly responsible for judging the technical quality of the objectives, evaluation methods, and content sequencing.
Factor 2 called Instructional Events, grouped items 1 and 2 of the scale, with factor loadings of 0.81 (Item 1) and 0.82 (Item 2), and alpha of 0.82. The suggestion to maintain the bifactorial structure, even with only 2 items in this second factor, was motivated by a theoretical implication that can collaborate with the proposition of new measures in Evaluation of educational actions at the Reaction level.
Discussion
The original scale called Reaction to Instructional Procedures in Distance Education was applied in several contexts, as already mentioned. In the study by Martins and Zerbini (2015); the authors found the presence of multicollinearity between two items – “Correspondence between the proposed content and the course objectives” and “Correspondence between the course content and the personal objectives”. In our study, these items are equivalent to the two items that alone composed Factor 2 – “Instructional events” (items “Connection between the subject and the objectives of my course” and “Connection between the subject and my personal objectives”), with a good internal consistency index (α = 0.82).
In the research by Martins et al. (2018), the instrument underwent a more rigorous refinement – the removal of 8 out of 17 items, in the 2015 scale. A large sample of workers (N = 3,600) from a Brazilian financial institution was considered in the new validation. Finally, the structure remained unifactorial (with only 9 items, α = 0.92) and presented a high level of adjustment in the Confirmatory Factor Analysis (Goodness-of-Fit Index – GFI = 0.95). However, according to the authors, the correlation between the errors of items 1 (“Correspondence between the proposed content and the course objectives”) and 2 (“Correspondence between the course content and the personal objectives”) was included. Martins et al. (2018) stated that, with the exception of the item “Correspondence between the course content and the personal objectives”, the remaining eight items on the scale referred to aspects of the course (planning) and not of the student. Thus, they understood that this may have led respondents not to evaluate “course objectives” and “personal objectives” differently. However, since both items have theoretical meaning, they were kept on the scale and the following research agenda was suggested: to test this instrument again, preferably in a different context with regard to previous studies.
It is worth noting that, in this research, the students were able to assess the difference between course objectives and personal objectives; more than 50% assessed the items between points 3 and 5 on the 7-point scale. The correlation between factors 1 and 2 of the instrument was 0.45, an index considered insufficient to indicate the existence of a general factor. The factor loading of the two variables above (1 and 2) was high (0.81 and 0.83, respectively) and both presented the greatest commonalities (h2 = 0.68 and h2 = 0.70), showing that they are well explained by Factor 2, which grouped them.
There is theoretical support for this bifactor structure. In the study by Martins and Zerbini (2015), which used the same scale; students also satisfactorily evaluated instructional procedures similar to the items in Factor 1 Instructional Planning. In the study of Giray (2021), Engineering students positively evaluated a good part of the teachers’ instructional procedures during emergency remote teaching. Considering that the authors conducted interviews with the students, more specific data were collected: 1) Some teachers recorded explanatory videos of specific content and students used these materials intensively; 2) Live classes were indicated by Engineering students as the best class format; 3) The methods used for assessment, according to students, should be adapted to online teaching, as well as the language and format of the course materials; 4) Students whose teachers constantly interacted with the class were more satisfied and motivated with the ERT.
Still in relation to teacher communication with students, research highlights the following negative factors in online teaching: superficial interaction with students in synchronous classes; teachers’ inability to use digital tools (Kebritchi et al., 2017); low levels of asynchronous “teacher presence” in the virtual environment; use of few interactive technologies that can effectively engage students in online learning (Iglesias-Pradas et al., 2021).
Naji et al. (2020), in interviews with engineering students, found that they responded satisfactorily to the constant interaction between teachers and their classes; they considered this procedure essential to help students understand the need and potential benefits of changing to ERT. According to the interviewees, continuous contact with teachers and peers increased motivation, appreciation for online learning, and helped reduce feelings of loneliness.
It is concluded about Factor 1 that, evaluating the students’ Reaction to the instructional planning meant, here, measuring their satisfaction with what (the program) and how (methods, teaching means) the course Calculus 2 was offered to them. In other words, the fact that the students reacted favorably to almost all teaching procedures means, specifically, that the students were satisfied with the instructional planning. However, it is essential to emphasize that this measurement is not sufficient to attest to the quality of the discipline program or its suitability to the course objectives. Proof of this is the analysis of the next factor that integrated the empirical structure of the scale in question.
Before indicating this implication, it is important to make this brief summary: (a) The Reaction to Instructional Procedures in the Distance Education scale was suggested after its first version that aimed to evaluate student satisfaction concerning instructional planning; (b) Instructional planning involves defining the content, sequence, and means by which the content will be transmitted to the student; (c) Measuring student reaction can also mean measuring teaching performance or the applicability/usefulness of the course or discipline (Meneses et al., 2010); (d) In Brazil, there are specific scales to measure the performance of tutors, graphical interface of courses, etc., as already mentioned; (e) The scale used here, Reaction to Instructional Procedures in Distance Education, aims to evaluate elements of the Planning and Execution subsystem, which integrates the TD&E system. This subsystem encompasses activities such as defining instructional objectives and content, teaching strategies (program sequence, choice of teaching resources), and appropriate evaluation criteria to achieve previously established objectives.
Continuing the reasoning, in Gagné’s theory, the author mentions that, for learning to occur, it is necessary to consider the student’s internal and external events. The internal events refer to the internal learning process that, although being a personal process, can and should be facilitated by the teacher. The external events, in turn, are the conditions provided to the student so that learning can occur.
Therefore, the elements that make up Gagné’s Instructional Events (1985) differ from those that make up the Instructional Planning, although both are complementary. It is this distinction that we wish to focus on, since they gave name to the factors of the empirical structure found in our investigation. In this sense, we can deduct that Instructional Planning encompasses procedures related to the design of the discipline, while Instructional Events lists the conditions for operationalizing this design.
Therefore, it is considered that the two variables of Factor 2 Instructional Events (“Link between the subject and the objectives of the course” and “Link between the subject and the personal objectives”) have greater adherence to Gagné’s theory. At the same time, the composition of this new factor points to the possibility of thinking about new measures based on the other instructional events. Evaluating these two variables in light of Gagné’s arguments means reviewing the students’ reaction to the way the subjects’ objectives were presented to them (i.e., the condition provided by the teacher) and not to the way they are written on “paper” (in the planning). In this reasoning about the new measure, the researcher would also evaluate the students’ reaction to the teacher’s performance (Getting the student’s attention, Informing the instructional objective, Presenting stimuli to the student) and to the applicability/usefulness of the subject. Therefore, it is proposed that Gagné’s instructional events be further explored as a theoretical basis for reviewing the items on the original scale Reaction to instructional procedures in distance learning or for suggesting new instruments that specifically contemplate this theory.
Conclusion
The ERT experience described in our investigation highlighted the urgent need for teachers to be updated on new tools, techniques, methods and taxonomies for in-person and online teaching and learning. This conclusion came not only from the empirical results of our investigation, but also from the literature review that was carried out. From now on, it is possible that online teaching becomes part of the pedagogical planning for Engineering courses worldwide – either as a complement to the in-person format, in the case of hybrid courses, or as an autonomous format, in the case of the development of new specific distance learning programs.
In order for institutions to be able to plan and/or evaluate their educational actions in practice, the instrument used in our investigation can be replicated in other contexts, considering other courses, including those in other areas of knowledge beyond the Exact Sciences. At the same time, when choosing the emergency remote teaching modality, the motivation was not only the urgency and novelty of the topic, but also the need to expand theoretical discussions on the paradigm of offering distance learning activities in Engineering courses in Brazil, especially in federal public HEI. In this case, the intention was also to contribute to studies on distance learning in Engineering.
The main limitation of our study was the failure to perform confirmatory factor analyses of the empirical structures. It is also worth noting that only one source of human information was used: the student. However, it is known that in order to involve other individuals and collect information from teachers and coordinators, for example, great efforts would be required, especially because the context was the pandemic and social distancing. Longitudinal studies could make it possible to include these data in the investigation design.
In the theoretical and methodological fields of the TD&E area, in our work we investigated the concepts related to Gagné’s (1985) instructional events. The objective, therefore, was to mature these theoretical proposals and advance methodologically with the suggestion of new items directly associated with the nine Instructional Events, in order to compose new psychometric scales that can improve research on evaluation measures of educational actions in the field of Engineering and other areas of knowledge.
-
How to cite this article:
Zerbini, T., & Kaizer, B. M. (2026) Adaptation and evidences of validity of the Remote Teaching Reactions Scale in Exact Sciences. Estudos de Psicologia (Campinas), 43, e230050. https://doi.org/10.1590/1982-0275202643e230050
Data availability
The research data are available within the body of the document.
References
- Abbad, G. S. (1999). Um modelo integrado de avaliação de impacto de treinamento no trabalho [Tese de doutorado não publicada]. Universidade de Brasília.
-
Abbad, G. S., Gama, A. L., & Borges-Andrade, J. E. (2000). Treinamento: análise do relacionamento da avaliação nos níveis de reação, aprendizagem e impacto no trabalho. Revista de Administração Contemporânea, 14(3), 25-45. https://www.scielo.br/scielo.php?pid=S1415-65552000000300003&script=sci_arttext
» https://www.scielo.br/scielo.php?pid=S1415-65552000000300003&script=sci_arttext -
Abreu, V. F., & Oliveira-Melo, F. G. (2020). Tecnologias educacionais utilizadas por monitores dos cursos de engenharia. Revista de Ensino de Engenharia, 39 http://revista.educacao.ws/revista/index.php/abenge/article/view/1636
» http://revista.educacao.ws/revista/index.php/abenge/article/view/1636 -
Alqurashi, E. (2019). Predicting student satisfaction and perceived learning within online learning environments. Distance Education, 40(1), 133-148. https://www.tandfonline.com/doi/abs/10.1080/01587919.2018.1553562
» https://www.tandfonline.com/doi/abs/10.1080/01587919.2018.1553562 -
Alsabahi, M. A., Bahador, K. M. K., Saat, R. M., & Ntim, C. G. (2021). The influence of personal characteristics and workplace learning on information technology competency among external auditors: The role of organisational culture as a moderator. Cogent Business & Management, 8(1). https://www.tandfonline.com/doi/full/10.1080/23311975.2021.1899625
» https://www.tandfonline.com/doi/full/10.1080/23311975.2021.1899625 -
Amaral, E., & Polydoro, S. (2020). Os desafios da mudança para o ensino remoto emergencial na graduação na Unicamp–Brasil. Linha Mestra, 41a, 52-62. http://lm.alb.org.br/index.php/lm/article/view/392
» http://lm.alb.org.br/index.php/lm/article/view/392 -
Assumpção, G., Hamada, P. C., & Castro, A. (2018). Análise do perfil dos alunos do primeiro curso de EaD em Engenharia de Produção em instituição de ensino superior pública. Revista Produção Online, 18(2), 404-423. https://producaoonline.org.br/rpo/article/view/2757
» https://producaoonline.org.br/rpo/article/view/2757 - Bloom, B. S., Krathwohl, D. R., & Masia, B. B. (1972). Taxonomia de objetivos educacionais, compêndio primeiro: domínio cognitivo Globo.
- Borges-Andrade, J. E. (2006). Avaliação integrada e somativa em TD&E. In J. E. Borges-Andrade, G. Abbad, & L. Mourão (Orgs.), Treinamento, desenvolvimento e educação em organizações e trabalho: fundamentos para a gestão de pessoas (pp. 343-358). Artmed.
-
Borsa, J. C., Damásio, B. F., & Bandeira, D. R. (2012). Adaptación y validación de instrumentos psicológicos entre culturas: Algunas consideraciones. Paidéia (Ribeirão Preto), 22(53), 423-432. https://www.scielo.br/j/paideia/a/cbRxjMqmbZddKpwywVM8mJv/?format=pdf⟨=pt
» https://www.scielo.br/j/paideia/a/cbRxjMqmbZddKpwywVM8mJv/?format=pdf⟨=pt -
Carlisle, J., Bhanugopan, R., & D’netto, B. (2019). Enhancing task performance through effective training: The mediating role of work environment and moderating effect of non-mandatory training. Journal of Business Research, 104, 340-349. https://doi.org/10.1016/j.jbusres.2019.07.033
» https://doi.org/10.1016/j.jbusres.2019.07.033 -
Carvalho, R. S., & Abbad, G. (2006). Avaliação de treinamento a distância: reação, suporte à transferência e impactos no trabalho. Revista de Administração Contemporânea, 10, 95-116. https://www.scielo.br/j/rac/a/vzNZjG54nNmnLB85KzckYWz/?format=pdf⟨=pt
» https://www.scielo.br/j/rac/a/vzNZjG54nNmnLB85KzckYWz/?format=pdf⟨=pt -
Dolch, C., & Zawacki-Richter, O. (2018). Are students getting used to learning technology? Changing media usage patterns of traditional and non-traditional students in higher education. Research in Learning Technology , 26, 20-38. https://journal.alt.ac.uk/index.php/rlt/article/view/2038
» https://journal.alt.ac.uk/index.php/rlt/article/view/2038 -
Eom, S. B., & Ashill, N. J. (2018). A system’s view of e-learning success model. Decision Sciences Journal of Innovative Education, 16(1), 42-76. https://onlinelibrary.wiley.com/doi/abs/10.1111/dsji.12144
» https://onlinelibrary.wiley.com/doi/abs/10.1111/dsji.12144 -
Ferreira, D. M., & Mourão, L. (2020). Panorama da educação a distância no ensino superior brasileiro. Revista Meta: Avaliação, 12(34), 247-280. http://dx.doi.org/10.22347/2175-2753v12i34.2318
» https://doi.org/10.22347/2175-2753v12i34.2318 - Gagné, R. M. (1985). The conditions of learning and theory of instruction (4th ed.). Holt, Rinchardt and Winston.
-
Giray, G. (2021). An assessment of student satisfaction with e-learning: An empirical study with computer and software engineering undergraduate students in Turkey under pandemic conditions. Education and Information Technologies, 26(6), 6651-6673. https://doi.org/10.1007/s10639-021-10454-x
» https://doi.org/10.1007/s10639-021-10454-x - Hamblin, A. C. (1978). Avaliação e controle do treinamento McGraw-Hill do Brasil.
-
Iglesias-Pradas, S., Hernández-García, A., Chaparro-Peláez, J., & Prieto, J. L. (2021). Emergency remote teaching and students’ academic performance in higher education during the COVID-19 pandemic: A case study. Computers in Human Behavior, 119, 106-713. https://www.sciencedirect.com/science/article/abs/pii/S0747563221000352
» https://www.sciencedirect.com/science/article/abs/pii/S0747563221000352 -
Kebritchi, M., Lipschuetz, A., & Santiague, L. (2017). Issues and challenges for teaching successful online courses in higher education: A literature review. Journal of Educational Technology Systems, 46(1), 4-29. https://doi.org/10.1177/0047239516661713
» https://doi.org/10.1177/0047239516661713 - Kirkpatrick, D. L. (1976). Evaluation of training. In R. L. Craig (Org.), Training and Development Handbook (pp. 18.1-18.27). Mc Graw-Hill.
-
Martins, L. B., & Zerbini, T. (2015). Evidências de validade de instrumentos de reações no ensino superior a distância. Estudos e Pesquisas em Psicologia, 15(1), 116-134. https://www.redalyc.org/pdf/4518/451844503008.pdf
» https://www.redalyc.org/pdf/4518/451844503008.pdf -
Martins, L. B., Zerbini, T., & Medina, F. J. (2018). Course reaction scale in e-learning: Adaptation and factor structure. Psicologia: Teoria e Prática, 20(1), 223-234. https://www.scielo.br/scielo.php?script=sci_arttext&pid=S0102-79722018000103107
» https://www.scielo.br/scielo.php?script=sci_arttext&pid=S0102-79722018000103107 -
Mayer, R. E. (2019). Thirty years of research on online learning. Applied Cognitive Psychology, 33(2), 152-159. https://onlinelibrary.wiley.com/doi/abs/10.1002/acp.3482
» https://onlinelibrary.wiley.com/doi/abs/10.1002/acp.3482 - Meneses, P. P. M., Zerbini, T., & Abbad, G. (2010). Manual de treinamento organizacional Artmed.
-
Murphy, L., Eduljee, N. B., & Croteau, K. (2020). College student transition to synchronous virtual classes during the COVID-19 pandemic in northeastern United States. Pedagogical Research, 5(4). https://eric.ed.gov/?id=EJ1275517
» https://eric.ed.gov/?id=EJ1275517 -
Naji, K. K., Du, X., Tarlochan, F., Ebead, U., Hasan, M. A., & Al-Ali, A. K. (2020). Engineering students’ readiness to transition to emergency online learning in response to COVID-19: Case of Qatar. EURASIA Journal of Mathematics, Science and Technology Education, 16(10). https://www.ejmste.com/download/engineeringstudents-readiness-to-transition-to-emergency-online-learning-in-response-to-covid-198474.pdf
» https://www.ejmste.com/download/engineeringstudents-readiness-to-transition-to-emergency-online-learning-in-response-to-covid-198474.pdf -
Nauman, S., Bathi, S., Jalil, F., & Riaz, M. B. E. (2020). How training at work influences employees’ job satisfaction: roles of affective commitment and job performance. International Journal of Training Research, 1-16. https://www.tandfonline.com/doi/abs/10.1080/14480220.2020.1864444
» https://www.tandfonline.com/doi/abs/10.1080/14480220.2020.1864444 -
Nguyen, K. D., Enos, T., Vandergriff, T., Vasquez, R., Cruz, P. D., Jacobe, H. T., & Mauskar, M. M. (2020). Opportunities for education during the COVID-19 pandemic. JAAD International, 1(1), 21-21. https://doi.org/10.1016/j.jdin.2020.04.003
» https://doi.org/10.1016/j.jdin.2020.04.003 -
Noe, R. A., & Schmitt, N. (1986). The influence of trainee attitudes on training effectiveness: Test of a model. Personnel Psychology, 39(3), 497-523. https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1744-6570.1986.tb00950.x
» https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1744-6570.1986.tb00950.x -
Pasquali, L. (2009). Psicometria. Revista da Escola de Enfermagem da USP, 43, 992-999. https://www.redalyc.org/pdf/3610/361033301002.pdf
» https://www.redalyc.org/pdf/3610/361033301002.pdf -
Peimani, N., & Kamalipour, H. (2021). Online Education and the COVID-19 Outbreak: A Case Study of Online Teaching During Lockdown. Education Sciences, 11(2), 72. https://doi.org/10.3390/educsci11020072
» https://doi.org/10.3390/educsci11020072 -
Phillips, J. M., & Gully, S. M. (1997). Role of goal orientation, ability, need for achievement, and locus of control in the self-efficacy and goal-setting process. Journal of Applied Psychology, 82(5). https://www.proquest.com/openview/f77762b9b6cf0c2f9becbb4877bd0d6a/1?pqorigsite=gscholar&cbl=60903
» https://www.proquest.com/openview/f77762b9b6cf0c2f9becbb4877bd0d6a/1?pqorigsite=gscholar&cbl=60903 -
Rapanta, C., Botturi, L., Goodyear, P., Guàrdia, L., & Koole, M. (2020). Online university teaching during and after the Covid-19 crisis: Refocusing teacher presence and learning activity. Postdigital Science and Education, 2(3), 923-945. https://doi.org/10.1007/s42438-020-00155-y
» https://doi.org/10.1007/s42438-020-00155-y -
Sade, P. M. C., Peres, A. M., Zago, D. P. L., Matsuda, L. M., Wolff, L. D. G., & Bernardino, E. (2020). Assessment of continuing education effects for nursing in a hospital organization. Acta Paulista de Enfermagem, 33(2), 1-8. https://acta-ape.org/wp-content/uploads/articles_xml/1982-0194-ape-33eAPE20190023/1982-0194-ape-33-eAPE20190023-en.pdf
» https://acta-ape.org/wp-content/uploads/articles_xml/1982-0194-ape-33eAPE20190023/1982-0194-ape-33-eAPE20190023-en.pdf -
Simms, L. J., Zelazny, K., Willians, T. F., & Bernstein, L. (2019). Does the number of response options matter? Psychometric perspectives using personality questionnaire data. Psychological Assessment, 31(4), 5-57. https://www.jite.org/documents/Vol15/JITEv15ResearchP157-190Sun2138.pdf
» https://www.jite.org/documents/Vol15/JITEv15ResearchP157-190Sun2138.pdf -
Sun, A., & Chen, X. (2016). Online education and its effective practice: A research review. Journal of Information Technology Education, 15, 157-190. https://hagamoshistoria.pe/uploads/file/OnlineEducationAndEffectivePractice.pdf
» https://hagamoshistoria.pe/uploads/file/OnlineEducationAndEffectivePractice.pdf - Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics Harper-Collins College Publishers.
- Warr, P., Bird, M., & Rackham, N. (1970). Evaluation of Management Training London.
-
Watermeyer, R., Crick, T., Knight, C., & Goodall, J. (2021). COVID-19 and digital disruption in UK universities: Afflictions and affordances of emergency online migration. Higher Education, 81, 623-641. https://doi.org/10.1007/s10734-020-00561-y
» https://doi.org/10.1007/s10734-020-00561-y -
Zerbini, T., & Abbad, G. (2008). Estratégias de aprendizagem em curso a distância: validação de uma escala. Psico-USF, 13(2), 177-187. https://www.scielo.br/j/pusf/a/7WDStVBjCsKkDmpSszBNkXq/?lang=pt
» https://www.scielo.br/j/pusf/a/7WDStVBjCsKkDmpSszBNkXq/?lang=pt -
Zerbini, T., & Abbad, G. (2009). Reação aos procedimentos instrucionais de um curso via internet: validação de uma escala. Estudos de Psicologia (Campinas), 26(3), 363-371. https://www.scielo.br/scielo.php?pid=S0103-166X2009000300009&script=sci_abstract&tlng=es
» https://www.scielo.br/scielo.php?pid=S0103-166X2009000300009&script=sci_abstract&tlng=es -
Zhang, W., Zhang, W., Wang, Y., Yang, L., & Wang, C. (2020). Suspending Classes without Stopping Learning: China’s Education Emergency Management Policy in the COVID-19 Outbreak. Journal of Risk and Financial Management, 13(5). https://econpapers.repec.org/article/gamjjrfmx/v_3a13_3ay_3a2020_3ai_3a3_3ap_3a55_3ad_3a332192.htm
» https://econpapers.repec.org/article/gamjjrfmx/v_3a13_3ay_3a2020_3ai_3a3_3ap_3a55_3ad_3a332192.htm
Edited by
-
Editor
Solange Muglia Wechsler
