The innovative engagement of external control with artificial intelligence: evaluation model

Abstract

Innovation, with the implementation and use of Artificial Intelligence (AI) tools, including generative AI (GIA), can have a positive impact on the performance of private or public entities, including Courts of Auditors. For these bodies, the optimization of their actions converges with their main objectives, especially those focused on oversight actions. Thus, this article proposes an evaluation model, composed of indicators with specific dimensions and criteria that can be included in the evaluation system used by Brazilian Courts of Auditors - called the Performance Measurement Framework for Brazilian Courts of Auditors (MMD-TC) -, which can also evaluate the use of innovations related to the use of artificial intelligence, including GAI, in the audits of the Courts of Auditors. To this end, we present research on the characteristics and advances in the audits of the Courts of Auditors using AI, indicating how the structure of the MMD-TC can support the process of engagement with innovation.

Keywords:
court of auditors; generative artificial intelligence; MMD-TC; evaluation model; assessment model

Resumo

A inovação, com a implementação e utilização de ferramentas de Inteligência Artificial (IA), inclusive a generativa (IAG), pode proporcionar impactos positivos no desempenho das entidades privadas ou públicas, nestas últimas inclusos os Tribunais de Contas. Para esses órgãos, a otimização de suas ações converge com seus principais objetivos, especialmente aos voltados para ações de fiscalização. Assim, o presente artigo propõe um modelo de avaliação, composto por indicador com dimensões e critérios específicos que podem ser incluídos ao sistema de avaliação utilizado pelos Tribunais de Contas brasileiros - chamado de Marco de Medição de Desempenho dos Tribunais de Contas do Brasil (MMD-TC) -, que também poderá avaliar o uso de inovações que se relacionam com a utilização de inteligência artificial, incluindo a IAG, nas fiscalizações dos Tribunais de Contas. Para tanto, é apresentada uma pesquisa sobre as características e os avanços nas fiscalizações dos Tribunais de Contas com o uso de IA, indicando como a estrutura do MMD-TC pode apoiar o processo de engajamento com a inovação.

Palavras-chave:
tribunal de contas; inteligência artificial generativa; MMD-TC; modelo de avaliação; modelo avaliativo.

Resumen

La innovación, mediante la implementación y el uso de herramientas de inteligencia artificial (IA), incluyendo la inteligencia generativa, puede tener impactos positivos en el desempeño de entidades privadas o públicas, incluyendo los tribunales de cuentas. Para estos organismos, optimizar sus acciones converge con sus principales objetivos, especialmente aquellos enfocados en acciones de fiscalización. Por lo tanto, este artículo propone un modelo de evaluación, compuesto por un indicador con dimensiones y criterios específicos que puede incluirse en el sistema de evaluación utilizado por los tribunales de cuentas brasileños, llamado Marco de Medición del Desempeño de los Tribunales de Cuentas de Brasil (MMD-TC), que también puede evaluar el uso de innovaciones relacionadas con el uso de inteligencia artificial, incluyendo la IAG, en las auditorías de los tribunales de cuentas. Para ello, se presenta un estudio sobre las características y avances en las auditorías de los tribunales de cuentas con el uso de IA, indicando cómo la estructura del MMD-TC puede apoyar el proceso de involucramiento con la innovación.

Palabras clave:
tribunal de cuentas; inteligencia artificial generativa; MMD-TC; modelo de evaluación; modelo evaluativo.

1. INTRODUCTION

Innovation is a driving force for promoting the development of a group, region, or organization. A clear example of this is the fourth industrial revolution, a concept created in Germany in 2011 to describe a new era of technological advances and improvements in production processes (Drath, 2014). This concept poses a challenge for institutions (Müller & Däschle, 2018), forcing them to rethink current business practices in order to adapt to new demands (Ibarra et al., 2018).

More recently, the emergence of OpenAI’s ChatGPT in 2022, a form of Generative Artificial Intelligence (GAI), revealed to the world the high speed of transformation to which we are subjected.

For public external control institutions, especially Courts of Auditors, there is a growing need to adapt to innovation processes by establishing more agile actions, such as those that use Artificial Intelligence (AI), including GAI.

Among their possible actions, inspections stand out, generally on the entity’s own initiative, evaluated and defined based on principles such as risk, materiality (Organização Internacional das Entidades Fiscalizadoras Superiores [Intosai], 2017), relevance, and opportunity (Tribunal de Contas da União [TCU], 2016). These characteristics make audits an appropriate approach for pursuing innovation.

Since 2013, these courts have undergone a biennial national assessment called the Performance Measurement Framework for Brazilian Courts of Auditors (MMD-TC), created on the initiative of the Association of Members of Brazilian Courts of Auditors (Associação dos Membros dos Tribunais de Contas do Brasil [Atricon]), which provides guidelines to support improvement processes in the Courts of Auditors and establishes criteria for evaluating the performance of these bodies, in line with the international standards of Intosai (International Organization of Supreme Audit Institutions).

Seeking to align the need for these courts to engage in the innovation ecosystem, this article aims to propose a model for evaluating innovation external control using AI, including IAG, in the audits of Courts of Auditors.

To this end, it will present research on the characteristics and advances in the audits of the Courts of Auditors using AI, in addition to addressing the evaluation of audits by the MMD-TC and how this structure can support the process of engagement with innovation.

2. BASIS FOR INNOVATION

The term “innovation” comes from the Latin innovare, which means to do something new. In economics, innovation is when you add value to a creative idea. Although creativity is essential for innovation, it alone is not enough, and the following requirements are necessary: novelty, even if incremental; practical application; and value addition (Figueiredo, 2023).

The need for innovation is increasingly present in the professional scenario, especially with the growing challenges faced by companies and other entities after the COVID-19 pandemic. In this sense, the importance of having an innovation ecosystem (IE) comes to the fore. According to Adner (2006), these are collaborative arrangements to enhance innovation and achieve the best offer, commonly equipped with information technology, which cannot be achieved by a single initiative, as shown below.

Figure 1 shows the need to properly identify what is expected from the innovation process, as well as the risks related to coordination, initiatives/solutions, and the integration of these solutions throughout the value chain. In short, knowing what you want to solve and what the expected scope is.

FIGURE 1
STRATEGY FOR IMPLEMENTING THE INNOVATION ECOSYSTEM

Also, according to Mariani et al. (2023), the adoption of AI for innovation suggests advances in economic results, in this case linked to performance, effectiveness, and efficiency; competitive and organizational results; and innovation results. Thus, artificial intelligence is suitable for promoting innovation, as it offers powerful tools to accelerate the development of new products, services, and business models.

3. THE MMD-TC AS A FRAMEWORK FOR EVALUATING EXTERNAL CONTROL AND THE CRITERIA FOR EVALUATING INSPECTIONS

The MMD-TC aims to provide guidelines for improving the performance of courts of auditors. This assessment takes place every two years and uses a questionnaire based on Intosai’s Supreme Audit Institutions - Performance Measurement Framework (SAI PMF) methodology.

Taking regional differences into account, the results of each court are presented separately, with general disclosure made in a consolidated manner.

This methodology is the main tool for evaluating Courts of Auditors in Brazil and includes guidelines from the Brazilian Public Sector Auditing Standards (NBASP), Atricon, and the International Standards of Supreme Audit Institutions (ISSAIs). In 2019, the methodology received ISO 17021 certification from the Vanzolini Foundation, the only Brazilian member of The International Certification Network (IQNet).

The MMD-TC not only measures performance statically, but also adapts to new needs and practices, reflecting its connection with innovation in the public sector.

The general structure of the MMD-TC is represented in Figure 2 below:

FIGURE 2
MMD-TC STRUCTURE

According to the MMD-TC Procedures Manual (Atricon, 2024), the assessment is divided into six domains, which together cover 25 indicators. Each indicator is divided into several dimensions, each with different assessment criteria.

According to the manual, the six domains that make up the performance measurement framework cover various aspects of performance. With regard to inspections, which are addressed in more than one domain, domain C covers the more general aspects of inspections and is therefore suitable for the inclusion of broader parameters.

4. METHODS AND PROCEDURES

The research adopted both a quantitative and qualitative approach. The qualitative part is based on the relationship between the objective and subjective worlds, in which information cannot be expressed numerically. On the other hand, quantitative research focuses on aspects that can be measured and expressed in numbers, allowing for the classification and analysis of data (Prodanov & Freitas, 2013).

With regard to the level of knowledge, this involves the development of an evaluation model.

Quantitative data were processed through tabulation in spreadsheets and statistical analysis. Semantic analysis was used for qualitative analysis.

The tools used in the study were based on Choguill’s (2005) matrix, as detailed in Box 1 below:

BOX 1
METHODOLOGY MATRIX

Objectives		Hypotheses	Method of analysis	Expected results
	Analyze and/or research courts of auditors that use AI, including generative AI, their characteristics, and their advances.	Obtain the characteristics of the work, as well as the advances achieved. Failure to identify the characteristics of this work or to ascertain the advances made.	Analysis of the results of the research conducted.	List the advances achieved. List the characteristics and criteria identified.
Evaluate possible criteria for MMD-TC, for inspections using AI.		No suitable criteria were identified for this type of evaluation. Suitable criteria for evaluation were identified.	Documentary analysis of the MMD-TC as an evaluation model. Analysis of current criteria in the context of audits.	Present an indicator with specific dimensions and criteria for the MMD-TC that relate to the use of AI, including IAG, in audits by the Courts of Auditors.

To construct the evaluation model, criteria were researched that can be used to characterize work that effectively uses artificial intelligence. Also, research was conducted on the audits of the Courts of Auditors using AI, including IAG, its characteristics, and its advances. Based on these results, it was possible to analyze the current MMD-TC evaluation model and list specific criteria related to the use of AI - including AGI - in audits.

4. DATA ANALYSIS AND DISCUSSION

4.1. Use of artificial intelligence by courts of auditors

The Information Technology (IT) Committee of the Courts of Auditors of the Rui Barbosa Institute (Instituto Rui Barbosa [IRB]) and Atricon conducted research on the adoption of AI in Brazilian Courts of Auditors for the years 2023 and 2024, which included aspects such as types of AI solutions implemented; stages of implementation, training, and human resources; integration with other systems; and collaboration with partner entities.

According to IRB and Atricon (2024, p. 17), the survey revealed that “60% of Courts of Auditors have already implemented AI solutions in areas of external control, such as audits and inspections.”

It also indicated that the main objectives of AI implementation by Courts of Auditors are: fraud detection, task automation (document review, compliance verification, large data volume analysis, etc.), cost reduction (associated with manual work and related to errors and rework), and improved efficiency (optimizing audit response time and increasing service quality). Box 2 below shows the evolution of the objectives of the Courts of Auditors over two years:

BOX 2
COMPARISON OF THE MAIN OBJECTIVES FOR THE IMPLEMENTATION OF AI IN THE COURTS OF AUDITORS IN 2023 AND 2024

Objective	2023	2024
Fraud detection	Initial focus on identifying fraud through data analysis.	Expansion of capabilities to include fraud prediction and prevention.
Task automation	Basic automation of repetitive processes and document review.	Expansion to automation of more complex processes and integration with other systems.
Cost reduction	Use of AI to save on manual processes and reduce human error.	Broader use of AI to optimize resources and reduce operating costs.
Improved efficiency	Improved speed and accuracy in audits and reviews.	Implementation of AI to improve efficiency across a wider range of process , including administrative areas.

With regard to the types of AI solutions and stages of implementation, Box 3, prepared from the database provided, contains the following information:

BOX 3
MAIN RESPONSES RELATED TO AI SOLUTIONS AND IMPLEMENTATION STA GES BY COURT OF ACCOUNTS

Question	Main responses
Does your institution have specific policies or guidelines for the implementation or use of AI?	67%: No; 27%: Under development.
What are the biggest barriers to implementing AI solutions?	More than one answer possible: 55%: Lack of technical knowledge; 55%: Difficulties in hiring; 30%: Institutional resistance; 36%: Challenges regarding information security and data privacy.
Has your Court of Auditors given the AI project a name (e.g., ChatTCU, ANIA, ContAI, VigIA, Aurora, AçAI, among others)?	67%: Yes; 27%: No; 6%: There is no project.
What is the stage of development of this project?	46%: In use or in pilot phase; 33%: Under development; 12%: No action currently; 9%: In planning.
How was this project developed by your court in terms of collaboration with other entities?	42%: In partnership (academia, private or public institutions); 39%: No collaboration or support from other entities.
Did the project make use of large language models, also known as LLMs (e.g., GPT-4, LLaMA, PaLM2, among others)?	More than one answer possible: 58% GPT; 18%: Did not respond; 15%: Did not use or not applicable; 6%: BERT; 3%: Gemini.
Was this project developed with the aim of integrating with an electronic process platform?	61%: Yes.

The box shows that the Courts of Auditors are maturing in their approach to AI. Although most have artificial intelligence projects, many of these projects are related to large language models (LLMs) such as GPT, with no specific policies or guidelines for the implementation or use of AI. Less than half of these courts carry out initiatives in collaboration with other entities.

The lack of technical knowledge or the difficulty of hiring personnel were presented as the main barriers to the implementation of AI solutions. In terms of technical training, IRB and Atricon (2024) advocate for continuous training programs in order not to limit the full use of AI tools.

As for the lack of guidelines, IRB and Atricon (2024) point out that without clear guidelines, courts will find it difficult to define technical and ethical requirements for AI hiring and to train auditors and other civil servants.

Regarding the need to work with strategic partnerships, what we see is that promoting them tends to enable courts of auditors to move toward innovative AI solutions that are still nonexistent or in their infancy, such as those that use active AI, which differs from passive AI in that it does not depend on the user’s interaction skills - being capable of performing several simultaneous tasks both to achieve operational objectives and to ensure important principles for institutions such as transparency, ethics, traceability, inclusion, human validation, and large-volume data processing (Bliacheriene & Araújo, 2024).

4.2. Evaluation model with indicators with dimensions and criteria for the MMD-TC suitable for evaluating inspections using artificial intelligence

Inspections represent a large part of the powers of the Courts of Auditors described in Article 71 of the Federal Constitution of 1988 (Constituição da República Federativa do Brasil de 1988), being directly present in items IV, V, and VI of that article and consistent with s with robust regulations developed in this regard - such as the INTOSAI manuals and the NBASP.

As these are actions in which freedom of planning and choice of actions predominate, it is considered beneficial to optimize the way in which these actions are carried out and, therefore, create an environment conducive to significant improvements in the performance of these bodies in relation to society - consistent with the main objectives listed by the Courts of Auditors for the implementation of AI, presented in Box 2 above.

Thus, it is appropriate to propose an evaluation model, suggesting the inclusion of an indicator composed of specific dimensions and criteria for the MMD-TC that are related to the use of AI, including IAG, in the audits of the Courts of Auditors. Thus, the model is included in dimension C of the MMD-TC, with the inclusion of one more evaluation indicator, as proposed in Box 4 below:

BOX 4
PROPOSED EVALUATION MODEL FOR THE MMD-TC RELATED TO THE USE OF ARTIFICIAL INTELLIGENCE, INCLUDING AGI, IN THE AUDITS OF THE COURTS OF AUDITORS

DIMENSION C: INSPECTION AND AUDIT
Item	Criterion	Example of evidence	Regulatory sources	Scoring rules
15A - Al solutions in inspections
15A.1	Guidelines and requirements for Al implementation and use
	The Court of Auditors has guidelines		Atricon
15A.1.1	or policies for the implementation of Al, with clear guidance to auditors on initiatives and use.	Regulatory act.	Resolution XX/
15A.1.2	The Court of Auditors has IAG solutions under development or in use for use in audits.	Supporting documentation, which may include dashboards, interaction environments, etc.	Atricon Resolution XX/	Score = 4; all criteria met; Score = 3; 3 criteria met; Score = 2; 2 criteria met; Score = 1; 1 criterion met;
15A.1.3	The Court of Auditors provides auditors who carry out audits with ongoing training in order to promote adequate skills in the full understanding and use of Al, including IAG, in audit processes.	Training plan and/ or documentation proving ongoing training.	Atricon Resolution XX/	Score = 0; no criteria met.
15A.1.4	The Court works collaboratively with academic, public, or private entities to expand the potential for innovation in audits provided by Al, including Active IAG initiatives.	Cooperation agreements, covenants, or other similar documents.	Atricon Resolution XX/ XXX
15A.2	Inspection processes using IAG
15A.2.1	The Court of Auditors supervises or audits using IAG solutions focused on fraud detection; automation of tasks involving large volumes of data; cost reduction due to manual work, errors, or rework; and improved efficiency through time reduction or quality improvement.	Inspection processes.	Atricon Resolution XX/ XXX
15A.2.2	The Court of Auditors monitors bidding with the implementation of IAG solutions.	Processes or procedures for monitoring bidding processes.	Atricon Resolution XX/ XXX	Score = 4; all criteria met; Score = 3; 3 criteria met; Score = 2; 2 criteria met;
15A.2.3	The Court of Auditors implements IAG solutions to monitor information provided by jurisdictions regarding public procurement or privatization processes.	Processes or procedures for monitoring or tracking.	Atricon Resolution XX/ XXX	Score = 1; 1 criterion met; Score = 0; no criteria met.
15A.2.4	The Court of Auditors implements IAG solutions to monitor information provided by those under its jurisdiction regarding public tenders, personnel management (payroll, hiring, and concessions), or the evaluation of other public policies.	Monitoring processes or proceduresortrails.	Atricon Resolution XX/ XXX

The evaluation model encourages the use of generative AI, including active IAG, in inspections focused on fraud detection; automation of tasks involving large volumes of data and operations; reduction of costs associated with manual work, errors, and rework; and improvement of efficiency by reducing time or increasing quality-in line with the results of item 4.1.

To this end, it also establishes as fundamental that the Courts of Auditors have clear guidelines for the implementation and use of AI by auditors, and that these auditors have adequate technical competence to use and analyze the results of AI solutions.

Finally, it proposes to enhance collaborative actions with other institutions, whether public or private, as a way to expand the potential for innovation and respond more quickly to current needs.

5. FINAL CONSIDERATIONS

In this article, we seek to present an evaluation model that aims to promote the adoption of innovation provided by artificial intelligence in its operational and administrative practices in the external control system.

Considering the evolutionary profile of the MMD-TC evaluation system, which provides for its frequent review to include adjustments and developments aimed at promoting the adoption of practices capable of improving the actions of the Courts of Auditors, the proposed model presents a theoretical survey and evaluation criteria ready for validation. This type of initiative is expected to encourage the adoption of this new dimension of evaluation and, thus, innovative practices using IAG will not only be identified but also stimulated and shared throughout the Brazilian external control system.

As part of a continuous improvement evaluation methodology, the suggestion is to conduct research that measures the results of this model and, in the future, studies and proposes new items to compose the evaluation.

Finally, we emphasize that fostering innovation through generative artificial intelligence aims to expand the potential for innovation and respond more quickly to the current needs of the external control system and Brazilian society.

• Peer review report:
  Reviewers:
  Marco Antonio Carvalho Teixeira (Fundação Getulio Vargas, São Paulo / SP - Brazil) https://orcid.org/0000-0003-3298-8183
  One reviewer did not authorize the disclosure of their identity.
  The peer review report is available at this link Publons
• [AI-assisted translated version] Note: All English quotes were translated by this article’s translator.

ACKNOwLEDgMENTS

We would like to thank the Court of Auditors of the State of Espírito Santo for its support, which enabled the author to participate in the MBA course in Auditing and Innovation in the Public Sector at USP, from which this article is derived. We would also like to thank the Rui Barbosa Institute, in partnership with the University of São Paulo, for preparing, planning, and offering the course.

DATA AVAILABILITY

The data supporting this study derive from anonymous responses to the questionnaire and contain sensitive institutional information; for this reason, they are not publicly available. Upon reasonable request to the authors, aggregated material that does not allow the identification of respondents may be provided.

References

• Adner, R. (2006, April). Match Your Innovation Strategy to Your Innovation Ecosystem. Harvard Business Review https://hbr.org/2006/04/matchyour-innovation-strategy-to-your-innovationecosystem
  » https://hbr.org/2006/04/matchyour-innovation-strategy-to-your-innovationecosystem
• Associação dos Membros dos Tribunais de Contas do Brasil. (2024). Manual de Procedimentos - MMD-TC, versão 1.0 https://qatc.atricon.org.br/wp-content/uploads/2024/03/Manual-de-ProcedimentosMMD-TC_versao-1.0_11.03.2024.pdf
  » https://qatc.atricon.org.br/wp-content/uploads/2024/03/Manual-de-ProcedimentosMMD-TC_versao-1.0_11.03.2024.pdf
• Bliacheriene, A. C., & Araújo, L. V. (2024). Inteligência artificial generativa e ativa: desvendando mitos e oportunidades Arténa.
• Choguill, C. (2005). The research design matrix: A tool for development planning research studies. Habitat International, 29(4), 615-626. https://doi.org/10.1016/j.habitatint.2005.06.001
  » https://doi.org/10.1016/j.habitatint.2005.06.001
• Constituição da República Federativa do Brasil de 1988 (1988). https://www.planalto.gov.br/ccivil_03/Constituicao/Constituicao.htm
  » https://www.planalto.gov.br/ccivil_03/Constituicao/Constituicao.htm
• Drath, R. (2014). Industrie 4.0 - eine Einführung Open Automation. https://www.smart-production.de/open-automation/news-detailansicht/nsctrl/detail/News/industrie-4-0-eine-einfuehrung
  » https://www.smart-production.de/open-automation/news-detailansicht/nsctrl/detail/News/industrie-4-0-eine-einfuehrung
• Figueiredo, P. N. (2023). Estratégia nacional de inovação: uma breve contribuição para sua efetividade sob a perspectiva de acumulação de capacidade tecnológica. Revista de Administração Pública, 57(5), e2022-0418. https://doi.org/10.1590/0034761220220418
  » https://doi.org/10.1590/0034761220220418
• Ibarra, D., Ganzarain, J., & Igartua, J. I. (2018). Business model innovation through Industry 4.0: A review. Procedia Manufacturing, 22, 4-10. https://doi.org/10.1016/j.promfg.2018.03.002
  » https://doi.org/10.1016/j.promfg.2018.03.002
• Instituto Rui Barbosa & Associação dos Membros dos Tribunais de Contas do Brasil. (2024). Inteligência Artificial nos Tribunais de Contas: Avanços e Desafios https://irbcontas.org.br/wp-content/uploads/2024/09/inteligencia-artificial-nos-tcsavancos-e-desafios-v2.pdf
  » https://irbcontas.org.br/wp-content/uploads/2024/09/inteligencia-artificial-nos-tcsavancos-e-desafios-v2.pdf
• Mariani, M. M., Machado, I., Magrelli, V., & Dwivedi, Y. K. (2023, April). Inteligência artificial na pesquisa de inovação: uma revisão sistemática, estrutura conceitual e futuras direções de pesquisa. Tecnovação, 122, 102623. https://doi.org/10.1016/j.technovation.2022.102623
  » https://doi.org/10.1016/j.technovation.2022.102623
• Müller, J. M., & Däschle, S. (2018). Business Model Innovation of Industry 4.0 Solution Providers Towards Customer Process Innovation. Processes, 6(12), 260. https://doi.org/10.3390/pr6120260
  » https://doi.org/10.3390/pr6120260
• Organização Internacional das Entidades Fiscalizadoras Superiores. (2017). ISSAI 100: Princípios fundamentais de auditoria do setor público Intosai. https://portal.tcu.gov.br/data/files/80/04/47/3A/C1DEF610F5680BF6F18818A8/ISSAI_100_principios_fundamentais_auditoria_setor_publico.pdf
  » https://portal.tcu.gov.br/data/files/80/04/47/3A/C1DEF610F5680BF6F18818A8/ISSAI_100_principios_fundamentais_auditoria_setor_publico.pdf
• Prodanov, C. C., & Freitas, E. C. (2013). Metodologia do trabalho científico: método e técnicas de pesquisa e do trabalho acadêmico (2a ed.). Feevale.
• Tribunal de Contas da União. (2016). Orientações para Seleção de Objetos e Ações de Controle

• Editor-in-chief:
  Alketa Peci (Fundação Getulio Vargas, Rio de Janeiro / RJ - Brazil) https://orcid.org/0000-0002-0488-1744

• Associate editor:
  Gabriela Spanghero Lotta (Fundação Getulio Vargas, São Paulo / SP - Brazil) https://orcid.org/0000-0003-2801-1628

Publication Dates

• Publication in this collection
  02 Feb 2026
• Date of issue
  2025

History

• Received
  21 Nov 2024
• Accepted
  17 July 2025