Modeling the COVID-19 vaccination process using the FRAM Method

Abstract

Vaccination is crucial in a primary care unit and relevant for society. This study aimed to map and analyze the COVID-19 vaccination process adopting Resilience Engineering concepts to identify possible sources of failure and recommend improvements. A case study was conducted in a Primary Health Care unit, and data were collected through semi-structured interviews in a cross-sectional study. The vaccination process was analyzed and modeled using the Functional Resonance Analysis Method (FRAM), which provided a greater understanding of the procedural flow and the identification of the system’s primary functions and points of variability. The results pointed out the couplings between functions that can generate high variability, which engineered improvements in the system’s resilience by proposing countermeasures for the situations identified as bottlenecks.

Key words:
FRAM; Vaccination; COVID-19; Health System Resilience; Primary Health Care

Resumo

A vacinação é um processo crucial em uma unidade de atenção básica e de extrema importância para a sociedade. Este estudo buscou mapear e analisar o processo de vacinação da COVID-19 a partir de conceitos da Engenharia de Resiliência, de modo a identificar possíveis fontes de falhas e recomendar melhorias. Foi realizado um estudo de caso em uma unidade de Atenção Primária à Saúde, e a coleta de dados ocorreu por meio de entrevistas semiestruturadas em um estudo transversal. O processo de vacinação foi analisado e modelado utilizando o método de análise da ressonância funcional (FRAM), o que proporcionou uma maior compreensão do fluxo processual e a identificação das principais funções e pontos de variabilidade do sistema. Os resultados apontaram os acoplamentos entre funções que podem gerar alta variabilidade, o que possibilitou engendrar melhorias da resiliência do sistema a partir da proposição de contramedidas de contorno para as situações identificadas como gargalos.

Palavras-chave:
FRAM; Vacinação; COVID-19; Resiliência em Sistemas de Saúde; Atenção Primária à Saúde

Resumen

La vacunación es un proceso crucial en una unidad de atención primaria y de extrema importancia para la sociedad. Este estudio pretendía mapear y analizar el proceso de vacunación COVID-19 utilizando conceptos de la Ingeniería de la Resiliencia, con el fin de identificar posibles fuentes de fallo y recomendar mejoras. Se realizó un estudio de caso en una unidad de Atención Primaria de Salud y se recogieron datos mediante entrevistas semiestructuradas en un estudio transversal. El proceso de vacunación se analizó y modelizó mediante el método de análisis de resonancia funcional (FRAM), que proporcionó una mayor comprensión del flujo del procedimiento y la identificación de las principales funciones y puntos de variabilidad del sistema. Los resultados mostraron los acoplamientos entre funciones que pueden generar una alta variabilidad, lo que permitió diseñar mejoras en la resiliencia del sistema proponiendo contramedidas para las situaciones identificadas como cuellos de botella.

Palabras clave:
FRAM; Vacunación; COVID-19; Resiliencia en Sistemas de Salud; Atención Primaria de Salud

Introduction

The pandemic caused by the novel coronavirus (SARS-CoV-2) represented a global challenge for health systems^1-3. Although Brazil has an important public health network, the COVID-19 pandemic caused a significant growth in hospitalizations, which put pressure on the reorganization of the Unified Health System (SUS)^4,5. Vaccination activities for COVID-19 represented an extraordinary labor in the preexisting vaccination routines in the National Immunization Program (PNI)⁶.

In Brazil, the SUS relies on the National Immunization Program Information System (SI-PNI), which primarily aims to plan decision-making regarding vaccination activities^7-9. The SI-PNI is an important technological innovation in managing immunization operations, but its use remains a challenge^7,9. Despite its several benefits for health care services, according to Silva et al.⁹, “some challenges are observed regarding its operationalization, completeness, and quality of the data collected (duplicity and underreporting)”. Another issue is the workload imposed on health teams by the pandemic and the different challenges in properly recording information, which ranges from the lack of equipment and team training to the prioritization of patient care over records, given the higher-than-usual number of appointments^1,10-12.

Previous research has identified flaws in the vaccination process regarding the availability and quality of COVID-19 vaccination data in Brazil^1,10,12. According to data presented in the technical note ‘Vaccination Transparency’¹², 74% of the questions related to the COVID-19 vaccination process have issues such as incomplete (37%), totally lacking (30%), or inconsistent (7%) information. In this sense, it is important to know the nuances of using the SI-PNI and how it is used in the PHC units’ vaccination rooms during the vaccination process.

Furthermore, Health Information Systems (SIS) often reflect the complexity of health care. Ferreira et al.¹³ state that “health activities, in general, are often characterized as complex sociotechnical systems, with non-linear interactions between human beings and technological or organizational artifacts”. Therefore, to identify and manage risks, resilience engineering is a good approach to address variability and its impacts on complex sociotechnical systems^14-16 since ensuring adequate immunization registration on the user’s vaccination card and in information systems is one of the ten steps published by the Ministry of Health for PHC workers to ensure expanded vaccination coverage in the units¹⁷.

Health systems’ resilience is the ability of these systems to adapt to provide uninterrupted health services with safety and quality. This ability must be developed continuously and not only when an extraordinary event occurs, such as the COVID-19 pandemic^18,19. In this research, the critical moment of the pandemic was used to model and analyze the vaccination process, seeking its variabilities and exposing opportunities for improvement.

Given the above, this study aims to map and analyze the main points of variability in the COVID-19 vaccination process based on resilience engineering concepts^14,19. The case study was adopted as the research strategy. Observations were performed in a vaccination room of a PHC unit and interviews were conducted with the responsible team’s members. Based on the data collected, the vaccine registration and application process was modeled using the Functional Resonance Analysis Method (FRAM), an effective tool for identifying and understanding the leading causes of failure propagation in sociotechnical systems based on coupling variabilities between functions^15,20.

The National Immunization Program Information System

The SIS is a data collection and processing tool that transforms data and information into knowledge. The collection of immunization data supports important decisions and actions for health management^21-24. The PNI was created in 1973 and gathers information on vaccination nationwide. It is responsible for the national vaccination policy and the national operationalization plan for vaccination against COVID-19²⁵.

The SI-PNI was implemented in 2010 and records information on each vaccinated person individually, by age group and vaccine doses administered and calculates vaccination coverage²³.

The immunization process during COVID-19 has been monitored and followed through records made in the SI-PNI, facilitating the planning and control of the immunobiologicals distributed. The registration can be made in ‘real-time’ online or after vaccination, as in non-computerized vaccination rooms, without Internet connectivity, or in extramural vaccination outside the vaccination room, for example, at home²⁵.

Immunization record errors are defined as typing errors and doses administered outside the current vaccination protocol or divergent immunological protocol. Thus, they are classified as immunization errors²⁶.

Functional Resonance Analysis Method (FRAM)

FRAM is considered an analysis method that reflects Resilience Engineering and Safety II thinking and seeks to understand how work adjusts to ‘unexpected’ situations and what the positive and negative consequences of such adjustments may be^27,28. The FRAM analysis²⁸ aims to describe or represent how a system should function to achieve its objectives (i.e., daily work) and understand how the possible variability of functions (alone or in combination) can affect how this happens.

Although FRAM was initially applied in accident analysis, it is now used as a modeling tool in various domains, including health^29-33.

Material and methods

The study was divided into two stages: (i) mapping and analyzing the COVID-19 vaccination process and (ii) FRAM modeling.

Stage 1: Mapping and analyzing the COVID-19 vaccination process

A cross-sectional qualitative approach was employed to map the event, which involved adopting a case study as a research strategy, as it is “an empirical investigation that investigates a contemporary event within its current context, especially when the boundaries between the event and the context are not clearly defined”³⁴.

A PHC unit in the northern zone of Rio de Janeiro was selected by convenience to analyze the security of data recording during the vaccination process due to two main characteristics: (i) The possibility of access and availability of employees to collaborate in implementing the research and (ii) the interest of the organization’s leaders in contributing to the research, after presenting the research already conducted and this research’s purposes.

The analysis unit considered all professionals who directly administer vaccines and register immunization regarding COVID-19 vaccination activities in the PHC unit. The team consisted of one nurse and one nursing technician. All selected professionals expressly consented to participate in the research.

Data and field notes were collected through face-to-face and individual interviews with professionals to identify potential system variability. The interviews were semi-structured and followed the questionnaire in the link https://doi.org/10.48331/scielodata.A82INX, which consisted of sociodemographic and technical questions that addressed functional mapping and contextualization of using the SI-PNI system. Each interview lasted approximately two hours, with no need for repetitions or recording of audiovisual material.

This study was conducted in the capital of Rio de Janeiro state in November 2021, when the first dose of the COVID-19 vaccine was administered to people aged 12 and over, and the booster dose was administered to the following cases: people aged 59 and over; patients with a high immunosuppression level (12 and over); healthcare professionals and workers who took the second dose up to 31/05/21; and people aged 18 and over who took the second dose more than five months earlier³.

Stage 2: FRAM modeling

FRAM²⁷ was adopted to model the vaccination process. The FRAM model aims to analyze how variability in function outputs can be combined to prevent their resonance, which can lead to undesirable results³⁵. Hollnagel²⁷ recommends that the purpose of the analysis be defined before applying FRAM since the modeling can be related to accident investigation (looking at past events, retrospective analysis) or safety assessment (looking at future events, prospective analysis).

FRAM is based on four principles or assumptions about how things occur^27,28. The principle of equivalence (of successes and failures): failure and success are equivalent, as they have the exact origin: the causes of adequate or inadequate functioning.

1. The principle of approximate adjustments states that people constantly adjust their activities to adapt them to their environment (considering resources, demands, opportunities, conflicts, and interruptions) so that the actions correspond to the established conditions.

2. Principle of emergent events: Variability alone is unlikely to be large enough to cause a failure or accident. However, variability in multiple functions can combine in unforeseen ways and cause disproportionate and non-linear effects.

3. Principle of functional resonance: When the cause-effect principle (causality) cannot be explained, functional resonance can describe and explain non-linear interactions and outcomes.

Once the purpose of the investigation was defined, the FRAM was applied, which consists of four stages^27,28, namely:

• The first stage identifies and describes the system’s important functions and characterizes each function using six essential characteristics (called aspects) (Figure 1): Input, Output, Preconditions, Resources, Time, and Control. Together, the functions constitute an FRAM model.

• The second stage characterizes the potential variability for each function in the FRAM model and the possible actual variability in one or more model implementations.

• The third stage is identifying functional resonance, which consists of determining the possibility of functional resonance based on the dependencies/couplings between functions, given their potential and actual variability. This stage aims to determine how a function’s variability can spread in the system and how it can combine with the variability of other functions.

• The fourth step consists of developing recommendations/countermeasures for monitoring and influencing variability, either by mitigating variability that may lead to undesirable results or increasing variability that may lead to desired results.

Figure 1
FRAM model function or activity aspects.

Results and discussion

Modeling the studied environment with FRAM: A Primary Health Care unit

The vaccination process includes the organization and operation of the vaccination room, conservation of immunobiologicals, and procedures for preparing and administering the vaccine. Nursing technicians are allocated to the immunization room and supervised by a Responsible Nurse (RN).

The vaccination room begins to operate when the emergency room opens. Before proceeding with vaccination, nursing technicians must sanitize their hands, check the cold room’s temperature, take inventory, and record the relevant data in the Order and Occurrence Book. Afterward, the cooler boxes must be prepared to reach a temperature between 2°C and 8°C, and then the most commonly used vaccines must be transferred into them.

After preparing the room, the nursing technician must receive the patient and begin the care. The nursing technician must request documentation (CPF or National Health Card (CNS) and vaccination card or booklet), register the dose of the vaccine administered (patient’s name, date of birth, and immunobiological), which must be personalized/individualized in the SI-PNI system, make the entry in the control book, and complete the vaccination certificate. The nursing technician must do all patient listening and vaccine preparation and administration. Once the process is complete, the certificate is handed over, and the patient is released.

Step 1 - Identifying essential functions

Interviews were conducted with professionals from the selected PHC unit to understand the event studied better, including one RN and one Nursing Technician. The first respondent is 28, has a technical course in nursing and a higher education degree in Nursing, and has worked in the PHC for six months but has been in the vaccination room as an RN for two months. The second respondent is 33 years old, has a technical course in Nursing and has worked in the PHC for three months, during which time he has worked in the vaccination room.

The team of professionals analyzed the setting and summarized the main activities throughout the vaccination process, from welcoming the patient to their release after the vaccine was administered. Based on the responses, we identified the main functions and characteristics of the vaccination process.

The questions asked focused on daily routines rather than specific successes and failures. Questions in the questionnaire that addressed function aspects were based on work already accepted in the related literature^20,28,36. Questions that addressed system aspects were adopted from the work of Ferreira et al.⁷. The questionnaire employed is provided in the link: https://doi.org/10.48331/scielodata.A82INX.

After data collection, a general model of the vaccination process containing the potential variability of the functions was developed. The FRAM Model Visualiser (FMV), version 2.2.0, was used, as shown in Figure 2 below. The link https://doi.org/10.48331/scielodata.A82INX, provides the textual representations of the mapped functions.

Figure 2
General model of the vaccination process, created in FMV v.2.2.0..

Figure 2 presents the FRAM model considering the following setting: vaccination room, morning shift, vaccination of the first dose of the COVID-19 vaccine in the general population, adult patient without allergies/comorbidities, not pregnant, postpartum, or lactating women. This is a typical setting in the area with the highest volume of patients, per data presented in the Municipal Immunization Plan³. Twenty-one functions were identified in the process, all human type, of which one is performed by the RN, represented by the gray color in the model, and the others are under the responsibility of the nursing technician, represented by the white color.

In the construction of the FRAM model, the graphical representation of each function is made by a hexagon, and the tool’s fundamental principle is that each input (Input, Precondition, Resource, Control, Time) - that is, the input aspects - for a function is an Output of another function²⁸. Each input aspect represents the inputs of a function circled in bold and/or by all the lines that arrive at the hexagon through the aspects I (Input), P (Precondition), R (Resource), C (Control), and T (Time). The outputs are represented by the Output (O) aspect, circled in bold and/or by the lines that depart from it.

Step 2 - Identifying the variability

Hollnagel et al.²⁸ recommend describing the variability of functions’ outputs in a simplified way regarding Timing and Precision. Thus, concerning Timing, an Output can occur too early, on time, too late, or not at all. In terms of Precision, an Output can be precise, acceptable, or inaccurate.

A sinusoidal wave in the model representation signaled the functions with the most significant potential variability (Figure 2). Chart 1 lists the FRAM model’s functions, outputs, and potential variability.

Chart 1
System functions.

Activity performed	Output	Potential Output Variability
Open room	Room opened	Very late: The vaccination room must be opened first thing in the morning. Acceptable
Hand hygiene	Hands cleaned	In time Precise
Check the cold room’s temperature	The cold room’s temperature	In time Acceptable: Conference errors may occur.
Perform the inventory	Inventory performed	In time Precise
Enter in the order and occurrence book	Registration of order and occurrences	In time Acceptable: Registration error may occur.
Prepare cooler boxes	Boxes prepared	Very late: Interruptions may occur while performing this activity. Inaccurate: There may be inaccuracy in the temperature, which should be between 2°C and 8°C.
Transfer vaccines to cooler boxes	Vaccines transferred	In time Acceptable: Errors may occur during transfer. The most commonly used vaccines must be transferred.
Monitor temperature	Temperature	In time Precise
Prepare inputs	Inputs prepared	In time Precise
Connect equipment	Equipment connected	In time Precise
Receive patient	Patient received	In time Precise
Start service	Service started	Very late: The person vaccinating is the same person who records the information in the system. Acceptable
Request documentation	Documentation	In time Acceptable: The patient may not have documentation, such as a vaccination card.
Consult the system	System consulted	Very late: Wait for the computer to become available or Internet connectivity. Acceptable
Perform anamnesis	Anamnesis performed	In time Precise
Register in the SI-PNI	Nominal/individualized record of the vaccine dose applied	Very late: Wait for the computer to become available or Internet connectivity. Inaccurate: Not every field in the record is filled in (e.g., adverse events after vaccination and losses of immunobiologicals) to speed up the process.
Register in the vaccination book	Vaccine registration	In time Acceptable: Registration error may occur.
Complete receipt	Receipt	In time Precise
Prepare vaccine	Vaccine prepared	In time Acceptable: An error may occur when preparing the immunobiological.
Apply vaccine	Vaccine applied	In time Precise
Deliver receipt	Receipt delivered	In time Precise

• Source: Authors, 2024.

In summary, of the 21 functions, 11 show real variability in time or precision. The 11 functions identified with variability in their Output were: <Open room>, <Check cold room’s temperature>, <Enter in the Order and Occurrence Book>, <Transfer vaccines to cooler boxes>, <Prepare cooler boxes>, <Start Service>, <Request documentation>, <Consult the system>, <Register in the SI-PNI>, <Register in the vaccination book> and <Prepare Vaccine>.

Step 3 - Identifying the Functional Resonance

The FRAM basis describes the functions that comprise an activity or process. However, the relationship between the functions is not described directly but indirectly through the relationships defined by the functions’ aspects. The common technical term for such relationships is coupling²⁸. FRAM can show how functions are coupled and the potential sources of more significant variability, besides determining which functions may be affected in a given situation³⁷. For example, in Figure 2, the functions <Register in the SI-PNI> and <Complete receipt> are interconnected since the Output of the <Register in the SI-PNI> function serves as input to the <Complete receipt> function, so the two functions are potentially coupled. This means that the variability of the Output of <Register in the SI-PNI> can affect the performance of <Complete receipt>. Furthermore, the <Register in the SI-PNI> function uses the Output of the <Request documentation> function as a Resource, which means that the variability of the Output of <Request documentation> can affect the performance of <Register in the SI-PNI>. For example, the service registration is discontinued if the patient does not submit documentation.

The number of aspects is analyzed by following the potential couplings between the functions step by step²⁸. To this end, the aspects are organized into two groups, which can be upstream or downstream, also identified as the upstream and downstream groups, respectively²⁰. Based on the mapping of functions in the previous section, Chart 2 indicates the couplings between functions through the coupling numbers (NAC) of the functions. Each function’s upstream and downstream aspects are added together to do this. The ‘upstream’ NAC group comprises the input, resource, control, precondition, and time aspects. The outputs form the ‘downstream’ NAC group (Chart 2).

Chart 2
Functions and numbers of couplings (NAC).

Activity performed	Upstream NAC	Downstream NAC
Open room	4	1
Hand hygiene	2	2
Check the cold room’s temperature	1	1
Perform the inventory	2	1
Enter in the order and occurrence book	3	4
Prepare cooler boxes	3	2
Transfer vaccines to cooler boxes	6	1
Monitor temperature	1	2
Prepare inputs	2	1
Connect equipment	2	1
Receive patient	1	1
Start service	5	1
Request documentation	3	4
Consult the system	3	1
Perform anamnesis	1	2
Register in the SI-PNI	4	1
Register in the vaccination book	2	1
Complete receipt	2	2
Prepare vaccine	2	1
Apply vaccine	3	1
Deliver receipt	2	1

• Source: Authors, 2024.

According to Hollnagel et al.²⁸ and Volken and Rosário²⁰, the greater the number of upstream aspects, the more significant the function’s homeostatic potential; that is, the greater the function’s capacity to receive inputs with unexpected variability and, given its resilience capacity, prevent deviations from propagating in the process. Thus, the functions <Transfer vaccines to cooler boxes> and <Start service> have the greatest homeostatic potential, presenting the most significant sum of upstream NACs. On the other hand, the greater the number of downstream aspects a function has, the greater its potential for propagating failures. The functions <Enter in the Order and Occurrence Book> and <Request documentation> had the most significant potential for propagating failures, as they are functions with the most significant sum of downstream NACs. Notably, the functions <Enter in the Order and Occurrence Book>, <Transfer vaccines to cooler boxes>, <Start service>, and <Request documentation> are functions with the highest NAC when adding both aspects (upstream and downstream). This information shows how these functions are central to the activity. These results corroborate the findings of Volken and Rosário²⁰ concerning identifying homeostatic potential functions and fault propagation and allowing the implementation of applications in resilience engineering.

Step 4 - Proposed countermeasures

After applying FRAM, we identified countermeasures to ensure more substantial control of the identified variability situations to monitor, mitigate, and even prevent possible adverse impacts. These countermeasures were grouped into human, technological, and organizational countermeasures:

• 1) Human countermeasures:

  • a) Increasing the number of professionals dedicated to the vaccination process. The professionals interviewed highlighted the importance of having at least two professionals in the vaccination room and that the activities be divided in such a way as to ensure that the information is recorded in the system during the service in all cases. This action aims to reduce the possibility of errors in the performance of functions such as <Open room> and <Start Service>, which are considered to be only “Too late” in accuracy.

  • b) Expanding the activities of the Responsible Nurse with the implementation of a routine of constant monitoring of the records made by the vaccination team and sensitizing the team about the importance of using the system. This action aims to reduce the possibility of errors in performing functions such as <Enter in the Order and Occurrence Book> and <Register in the vaccination book>, which have only an “Acceptable” accuracy.

  • c) Correctly fill in the fields of the SI-PNI system, especially the data on losses of immunobiologicals. This action aims to reduce the possibility of errors in the performance of functions such as <Register in the SI-PNI>, whose accuracy is considered “Too late” and “Inaccurate”.

• 2) Technological countermeasures:

  • a) The SI-PNI should replace the permanent file with the control card. The replacement will allow better data management and increased monitoring of activities. It is highly recommended to eliminate parallel records and adopt a unified and integrated solution. This action aims to reduce the possibility of errors in performing functions such as <Request documentation>, whose accuracy is considered only “Acceptable”.

  • b) Vaccination units without an Internet connection can adopt a local network registration system and synchronize their records with the central SUS server. The synchronization should not exceed 48 hours to ensure data is available nationwide. This action aims to reduce the possibility of errors in the performance of functions such as <Consult the system>, whose accuracy is considered only “Too late”.

• 3) Organizational countermeasures:

  • a) Investing in a highly available technology infrastructure can mitigate the current access problems faced by the unit. The data collection instruments (computers, tablets, and cell phones) used to feed the information systems must be made available in adequate amounts to the team responsible for the vaccination process, as pointed out by the professionals during the interviews. This action aims to reduce the possibility of errors in the performance of functions such as <Consult the system>, whose accuracy is considered only “Too late”, and <Register in the SI-PNI>, whose accuracy is considered “Too late” and “Inaccurate”.

Creating an integrated and reliable database for the vaccination process can facilitate the detection of cases in which the system may become overloaded. Examples include identifying the need to purchase new doses of a vaccine, the side effects of vaccines on the local population, and better organization and planning of vaccination campaigns.

Despite the countermeasures presented, Walker et al.³⁸ state that investing in innovation and skills is necessary, stakeholders should agree on the direction to take, and access to inputs, financial resources, and infrastructure should be available to maintain a balanced sociotechnical system.

Complex sociotechnical systems, such as the PHC unit environment, have particularities that require a robust approach to solve their problems. FRAM models have been widely used to represent the variability that affects the functions of complex sociotechnical health systems^15,18,19.

This study’s results are consistent with previous research^13,29 and indicate that applying the FRAM model in health systems can identify sources of variability that contribute to functional resonance. As a result, the FRAM model improves the understanding of the problem based on the variability analysis. This analysis and implementation of corrective actions can curb resonance and increase the system’s resilience.

Conclusions

Vaccination is a crucial process in a PHC unit and extremely important to society. This study aimed to understand how FRAM can reduce registration errors in the COVID-19 vaccination room. The method was adopted to model and analyze the vaccination process to improve its resilience.

The model was applied in a real case study to analyze the security of data recording during the vaccination process. From the instantiated model, we identified potential sources of recording errors in the COVID-19 vaccination room, providing a greater understanding of the domain and identifying the system’s primary functions and points of variability. A positive point of FRAM is the method’s easy application, which requires a good understanding of how the work is performed without demanding more time than is acceptable or an unfeasible number of people involved.

The results indicated the coupling of functions that can generate high variability and countermeasures proposed. FRAM’s application provided the study participants with a didactic view of the evaluated process, facilitating the identification of discrepancies between the work as performed and the work as imagined. We also identified that, for the national vaccination campaign against COVID-19, the dose registration applied is nominal/individualized, which is the process most crucial activity.

The research shows its relevance and originality by proposing an explanatory design for the flow of information on COVID-19 vaccination.

It could serve as a roadmap for managing similar events in the future and as a basis for improving the resilience of the vaccination process. Future research development will integrate technological and organizational aspects besides human ones so that it is applicable in other healthcare situations or environments.

Acknowledgments

The authors are grateful to the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq), the Fundação Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES), the Fundação Carlos Chagas Filho de Amparo à Pesquisa do Estado do Rio de Janeiro (FAPERJ), and the Fundação Oswaldo Cruz (Fiocruz).

References

• ¹ Brasil. Ministério da Saúde (MS). Plano Nacional de Operacionalização da Vacinação Contra a COVID-19. Brasília: MS; 2021.
• ² Caetano R, Silva AB, Guedes ACCM, Paiva CCN, Ribeiro GR, Santos DL, Silva RM. Desafios e oportunidades para telessaúde em tempos da pandemia pela COVID-19: uma reflexão sobre os espaços e iniciativas no contexto brasileiro. Cad Saude Publica 2020; 36(5):e00088920.
• ³ Rio de Janeiro. Secretaria Municipal de Saúde. Plano Municipal de Imunização: Documento Técnico Vacinação contra a covid-19. Rio de Janeiro: Secretaria Municipal de Saúde; 2021.
• ⁴ Duarte SJS, Justino LCL, Torres NL, Valle SM, Stolte VP. Manual de capacitação para vacinadores contra a COVID-19. Coren MS 2021; 1:1-331.
• ⁵ Ramos TCS, Silva TF. O trabalho na Atenção Primária em Saúde e a pandemia por Covid-19: um relato de experiência. Res Soc Develop 2021; 10(3):e38210313396.
• ⁶ Castro D. Medidas de prevenção e controle para profissionais na vacinação contra covid-19 [Internet]. Cofen; 2021 [acessado 2021 out 29]. Disponível em: http://www.cofen.gov.br/medidas-de-prevencao-e-controle-para-profissionais-na-vacinacao-contra-covid-19_84612.html
  » http://www.cofen.gov.br/medidas-de-prevencao-e-controle-para-profissionais-na-vacinacao-contra-covid-19_84612.html
• ⁷ Ferreira AP, Silva BS, Pereira MAD, Oliveira VC, quites HFO, Amaral GG, Guimarães EAA. Checklist para avaliação do desempenho do Sistema de Informação de Imunização: desenvolvimento e validação. Rev Cuba Informa Cien Salud 2021; 32(1):1-18.
• ⁸ Sato APS. National Immunization Program: Computerized System as a tool for new challenges. Rev Saude Publica 2015; 49:39.
• ⁹ Silva BS, Souza KC, Souza RG, Rodrigues SB, Oliveira VC, Guimarães EAA. Structural and procedural conditions in National Immunization Program Information System establishment. Rev Bras Enferm 2020; 73(4):e20180939.
• ¹⁰ Fundação Oswaldo Cruz (Fiocruz). Qualidade dos dados de vacinação nas unidades de saúde de atendimento para Covid-19 [Internet]. 2021 [acessado 2021 out 29]. Disponível em: https://bigdata-covid19.icict.fiocruz.br/nota_tecnica_20.pdf
  » https://bigdata-covid19.icict.fiocruz.br/nota_tecnica_20.pdf
• ¹¹ Jornal da USP. Pesquisadores apontam problemas nos dados sobre vacinação de covid-19 no Brasil. São Paulo: Jornal da USP; 2021.
• Open Knowledge Brasil, Transparência Brasil, Transparência Internacional. TransparêEncia da Vacinação: Uma análise de organizações da sociedade civil [Internet]. 2021 [acessado 2021 out 29]. Disponível em: https://www.ok.org.br/wp-content/uploads/2021/03/Nota_Tecnica_Vacinacao_04032021.pdf
  » https://www.ok.org.br/wp-content/uploads/2021/03/Nota_Tecnica_Vacinacao_04032021.pdf
• ¹³ Ferreira DMC, Saurin TA, Rosso CB. O uso de Functional Resonance Analysis Method (Fram) na Análise de Integração entre Sistemas: Um Estudo no Contexto da Saúde. Anais do IX Congresso Internacional de Conhecimento e Inovação 2019; 1(1):1-15.
• Righi AW, Saurin TA. Engenharia de resiliência: um panorama de seus estudos e perspectivas de pesquisas futuras. In: XXXI ENCONTRO NACIONAL DE ENGENHARIA DE PRODUCAO. ABEPRO; 2011. p. 1-13.
• ¹⁵ McGill A, Smith D, McCloskey R, Morris P, Goudreau A, Veitch B. The Functional Resonance Analysis Method as a health care research methodology: a scoping review. JBI Evid Synth 2022; 20(4):1074-1097.
• ¹⁶ Santos LJ, Ribeiro SA, Schimitz EA, Silva MF, Alencar AJSM. Risk Factors in the Software Deployment Phase: A Case Study Applied in two Brazilian Government Companies. J Inform Systems Technol Manag 2022; 19:e202219015.
• ¹⁷ Souza PA, Gandra B, Chaves ACC. Experiências sobre Imunização e o Papel da Atenção Primária à Saúde. APS Rev 2020; 2(3):267-271.
• ¹⁸ Carvalho PVR, Bellas H, Viana J, Nunes PC, Arcuri R, Fonseca VS, Carneiro APM, Jatobá A. Transformative dimensions of resilience and brittleness during health systems' collapse: a case study in Brazil using the Functional Resonance Analysis Method. BMC Health Serv Res 2023; 23(1):349.
• Jatobá A, Carvalho PVR. Resiliência em saúde pública: preceitos, conceitos, desafios e perspectivas. Saude Debate 2022; 46(n. esp. 8):130-139.
• ²⁰ Volken FGB, Rosário CR. The use of resilience engineering tools for fault propagation mitigation in sociotechnical systems. Rev Gest Prod Oper Sistemas 2021; 16(2):39.
• Brasil. Constituição da República Federativa do Brasil de 1988. Diário Oficial da União 1988; 5 out.
• ²² Starfield B. Atenção Primária - Equilíbrio entre necessidades de saúde, serviços e tecnologia. Brasília: UNESCO, Ministério da Saúde; 2002.
• ²³ Brasil. Ministério da Saúde (MS). Plano Diretor de Tecnologia da Informação. DATASUS. Brasília: MS; 2016.
• ²⁴ Lima Filho SP, Silva MF, Oliveira J, Ruback L. Study about Gathering Features in Depression Detection' Problem with Health Professionals Community. iSys 2022; 15(1):1-10.
• ²⁵ Brasil. Ministério da Saúde (MS). Plano Nacional de Operacionalização Da Vacinação Contra a Covid-19. Vol 14. Brasília: MS; 2021.
• Brasil (GT-AINFO). Relatório de erros de registro [Internet]. 2016 [acessado 2021 ago 28]. Disponível em: http://pni.datasus.gov.br/sipni/Erros%20de%20Registro%20040516.docx.
• Hollnagel E. FRAM: The Functional Resonance Analysis Method: Modelling Complex Socio-Technical Systems. CRC Press; 2012.
• Hollnagel E, Hounsgaard J, Colligan L. FRAM-the Functional Resonance Analysis Method-a Handbook for the Practical Use of the Method. 2014.
• ²⁹ Roberto AC, Costa Júnior AL, Costa Júnior MM. Aplicação do método FRAM para análise da baixa cobertura vacinal do HPV no município de São José dos Campos. In: Simpósio de Pesquisa Operacional e logística da Marinha. São Paulo: Editora Blucher; 2020.
• ³⁰ Wachs P. Modelo para integração entre melhoria de procedimentos operacionais padronizados e capacitação de operadores de sistemas sócio-técnicos complexos. Porto Alegre: UFRGS; 2016.
• Petrillo A, Felice F, Petrillo L. The Analytic Functional Resonance Analysis to Improve Safety Management. In: Operations Management - Emerging Trend in the Digital Era. IntechOpen; 2021.
• ³² Arcuri R, Bellas HC, Ferreira DS, Bulhões B, Vidal MCR, Carvalho PVR, Jatobá A, Hollnagel E. On the brink of disruption: Applying Resilience Engineering to anticipate system performance under crisis. Appl Ergon 2022; 99:103632.
• ³³ Carvalho EA, Gomes JO, Jatobá A, Silva MF, Carvalho PVR. Employing resilience engineering in eliciting software requirements for complex systems: experiments with the functional resonance analysis method (FRAM). Cognit Technol Work 2021; 23(1):65-83.
• ³⁴ Yin RK. Estudo de Caso: Planejamento e Métodos. 2ª ed. Porto Alegre: Bookman; 2001.
• ³⁵ Carvalho PVR. The use of Functional Resonance Analysis Method (FRAM) in a mid-air collision to understand some characteristics of the air traffic management system resilience. Reliab Eng Syst Saf 2011; 96(11):1482-1498.
• ³⁶ Fogaça LB. Tomada de Decisão e Equilíbrio de Metas Conflitantes No Gerenciamento de Interrupções de Voo Em Empresa de Transporte Aéreo Regula [dissertação]. Porto Alegre: PUCRS; 2015.
• ³⁷ Priori FR, Saurin TA. Aplicação do fram para solução de problemas em sistemas sócio-técnicos complexos: estudo de caso em uma unidade hospitalar. Rev Prod Online 2019; 19(1):102-128.
• ³⁸ Walker B, Gunderson L, Kinzig A, Folke C, Carpenter S, Schultz L. A Handful of Heuristics and Some Propositions for Understanding Resilience in Social-Ecological Systems. Ecol Soc 2006; 11(1):art13.

• Chief editors:
  Maria Cecília de Souza Minayo, Romeu Gomes, Antônio Augusto Moura da Silva

Publication Dates

• Publication in this collection
  20 June 2025
• Date of issue
  June 2025

History

• Received
  29 Mar 2024
• Accepted
  24 Feb 2025
• Published
  26 Feb 2025