Conditions for outcomes evaluation in public policies: a debate on the role of institutionality

Condiciones para la evaluación de resultados en políticas públicas: un debate sobre el papel de la institucionalidad

Abstract

The goals of this analysis are (i) to offer elements of reflection to the actors who design and implement public policies; (ii) to keep the debate on outcomes evaluation alight; (iii) to contribute toward integrating the debate on evaluation to the complexity of actions in the public sector. This theoretical essay researched works published on the subject from 1979 to 2019. The investigation of the approaches offered by the selected authors allowed identifying variables that are relevant to the analysis of public policies. The variables are expressed in the influences of institutional trajectories, actors, and organizational context. The analysis brings the following concluions: (i) the link between institutionality and the capacity to conduct results evaluations in public policies deserves attention in future studies; (ii) the debate on the topic can be strengthened by analyses that consider not only the decisions adopted, but also the rules, norms and strategies that define the political-institutional scene in which public policies are implemented.

Keywords:
institutionality; public policy; outcomes evaluation

Resumo

A análise teve os objetivos de oferecer elementos de reflexão aos atores que desenham e implementam políticas públicas e de manter aceso o debate sobre a avaliação de resultados, concorrendo para integrá-lo à complexidade da ação no setor público. O estudo foi motivado a partir do levantamento de trabalhos sobre o tema, publicados no período de 1979 a 2019. A exploração das abordagens oferecidas pelos autores selecionados permitiu a identificação de variáveis que possuem relevância para a análise de políticas públicas e que se expressam nas influências exercidas pelas trajetórias institucionais, pelos atores e pelo contexto organizacional. Conclui-se que: (i) a vinculação entre a institucionalidade e a capacidade de condução de avaliações de resultados em políticas públicas merece ser objeto de atenção em estudos futuros; (ii) o debate sobre o tema poderá ser fortalecido por análises que considerem não apenas as decisões adotadas, mas também as regras, normas e estratégias, que definem a cena político-institucional na qual são implementadas as políticas públicas.

Palavras-chave:
institucionalidade; políticas públicas; avaliação de resultados

Resumen

Los objetivos de este análisis fueron: (i) ofrecer elementos de reflexión a los actores que diseñan e implementan políticas públicas; (ii) mantener vigente el debate sobre la evaluación de resultados; (iii) contribuir a integrar el debate sobre la evaluación y la complejidad de las acciones en el sector público. El ensayo teórico investigó trabajos relacionados desde 1979 hasta 2019. La exploración de los enfoques ofrecidos por los autores seleccionados permitió la identificación de variables que son relevantes para el análisis de políticas públicas y que se expresan en las influencias ejercidas por las trayectorias institucionales, los actores y el contexto organizacional. Se concluye que: (i) el vínculo entre la institucionalidad y la capacidad para llevar a cabo evaluaciones de resultados en políticas públicas merece atención en estudios futuros; (ii) el debate sobre el tema puede fortalecerse mediante análisis que consideren no solo las decisiones adoptadas, sino también las reglas, normas y estrategias que definen el escenario político-institucional en el que se implementan las políticas públicas.

Palabras clave:
institucionalidad; políticas públicas; evaluación de resultados

1. INTRODUCTION

This essay sought to examine how literature has dealt with the relations between the institutionality of public policy and the adoption of processes aimed at their outcomes evaluation. In the long run, the objectives of the work are to keep the debate on the subject of outcomes evaluation in public policies alight and to contribute towards integrating the analytical aspects presented here with the complexity of the actions in the public sector.

The survey carried out in the literature on public policy evaluation indicates that the topic has remained relevant in the public administration debate over the last four decades. The understanding that evaluation use is relevant both for the orientation and reorientation of public policies and for the measurement of governmental action in regards to the demands of society presents itself as a tendency in most Western societies (Anderson, 2008; Batista & Domingos, 2017; Bovaird, 2014; Bourgeois & Cousins, 2013; Cousins, Goh, Elliot, & Bourgeois, 2014; Crumpton, Medeiros, Ferreira, Sousa, & Najberg, 2016; Cunill-Grau & Ospina, 2012; Hall, 2004; Hanberger, 2011; Hill & Hupe, 2005; Labin, 2014; Lane, 2014; March, 2009; Mark & Henry, 2004; Mastenbroek, Voorst, & Meuwese, 2015; Meny & Thoenig, 1992; Oliveira & Passador, 2019; Oliveira & Pinho, 2011; Oliver, Innvar, Lorenc, Woodman, & Thomas, 2014; Oliver, Lorenc, & Tinkles, 2019; Patton, 1988, 1999; Pressman & Wildavsky, 1984; Ramos & Schabbach, 2012; Richardson, 2013; Sabatier & Weible, 2007; Scott, 1998; Serapioni, 2016; Suarez-Balcazar & Taylor-Ritzler, 2013; Vedung, 1997; Wandersman, 2014; Weiss, 1979, 1993, 1998, 1999, 2010; Winter, 1990).

There are several reasons given by the authors consulted to explain the importance of evaluation actions in the cycle of public policies and their influence on the definition of the agenda, formulation, and implementation of policies. Mark, Henry, and Julnes (2000) have identified that assessments should be made with the aim of providing the means for recognition of merit and value; for organizational and programmatic improvement in the governmental environment; and for the supervision and development of public policy knowledge. Anderson (2008), Mark and Henry (2004), Meny and Thoenig (1992) and Weiss (1988a) consider that the rationale for evaluation lies in the link between the evaluation processes and the search for social improvement, given that they offer government agents the opportunity to adjust policies in order to improve them for the benefit of society (social betterment).

However, in spite of the presence and relevance of the topic in the literature, Weiss (1999) stimulated the debate when analyzing contexts of implementation of public policies and identified contradictions between the recognition in the literature about the contribution of evaluative processes and their depreciation by public policy managers. The analysis conducted by Weiss (1999) linked the discredit and fragility of evaluation processes to the most prevalent models of governmental decision-making processes, and to the attributes of the public agents that operate in the governmental arena in which public policies are formulated and conducted.

Weiss (1999) identified important variables for the understanding of the relationships that are established in scenarios of public policy formulation: contexts with a high level of informality of the policy and with low level of debate would be associated to the low valuation of evaluation actions. In informal contexts, Weiss (1999) observed the tendency for the valuation of knowledge inherent to policymakers to the detriment of advice drawn from evaluation processes; while the decentralized, democratic and participatory contexts were associated with the valorization of the evaluation actions.

Weiss (1999) compared the ease of identifying the values associated with evaluation processes to the barriers faced in their entry into the public policy arena, noting that although evaluation actions have much to offer to public policy managers, they rarely base new policies on evaluation results. As stated by Weiss (1999) this contradiction in the context of governmental management can be attributed to the pressure exerted by competitiveness, as well as to the ideological factors and the fragmentation in the scenarios of implementation of public policies.

In the light of the analytical elements raised by Weiss’s work (1999, 1998, 1993, 1979), the following questions were asked: Are more institutionalized public policies more likely to have their results evaluated? What variables can be used to evidence the institutionality of a public policy?

The method used to conduct the study consisted of a systematic review on the subject of the evaluation of results in public policies, both on Brazilian and international bases, using search terms in Portuguese (avaliação de resultados; avaliação de impacto; políticas públicas; institucionalidade) and English (evaluation of results, impact evaluation, public policy, institutionality) for the period ranging from 1979 to 2019.

Papers which adopted the neo-institutionalist theoretical and methodological framework to address the factors involved in the evaluation of results in public policies were selected for the accomplishment of this analysis.

2. THE CONCEPTUAL BASIS OF DEBATE

In the field of public administration, the action of evaluating involves aspects that place it between science and art. It is, therefore, a reflection organized both on the content of public policies, and on their context of implementation. In this sense, evaluation implies providing analyses on the conditions that exist for the operation of a policy, the objectives that were achieved and the expected and unexpected effects that were produced on the reality that it sought to transform (Anderson, 2008; Bovaird, 2014; Hall, 2004; Hansen, Klejsntrup, & Andersen, 2013; Hill & Hupe, 2005; Mark et al., 2000; Meny & Thoenig, 1992; Oliver et al., 2019; Scott, 1998; Trosa, 2001).

Among the studies considered classic for the study of public policies, Pressman and Wildavsky (1984) related the existing interfaces between the evaluation of public policies, organizational learning and implementation. According to those authors (Pressman & Wildavsky, 1984), the evaluation broadens organizational learning and learning guides implementation. Thus, organizational learning would contain the essence for the understanding of the variables involved in the implementation and evaluation processes of public policies. Pressman and Wildavsky (1984) emphasized the role of the evaluation of public policies for the conduct of processes of adaptation and of political-institutional learning. In this sense, they highlighted the potential of evaluative actions to reconcile the knowledge obtained about the formulation and implementation of a given policy with the political-institutional power, generating possibilities for changing directions in the governmental arena.

In these terms, besides representing opportunities for learning and improving the organizational capacities involved in the implementation of public policies (Cousins et al., 2014; Mark & Henry 2004; Patton, 1988; Cousins & Leithwood, 1986), evaluation processes guide public managers on the quality and importance of the public policies implemented under their responsibility, providing them with information and elements of analyses (feedbacks) capable of contributing to improve, as a whole, the organizational functioning of administrative structures. As maintained by Anderson, (2008), Trosa, (2001), Mark and Henry (2004) and Mark et al. (2000) evaluation actions are fundamental tools to increase the transparency and visibility desired by democratic states, especially in the contexts that value the systematic institution of accountability to society as an exercise of citizenship.

Weiss (1999) placed citizens as the primary beneficiaries of evaluation actions, defining that the central objective of evaluation processes is to help public organizations improve their policies, plans and practices, always in the interest of society.

For Mark and Henry (2004) evaluation actions are linked with the way people attribute meaning to public policies, which are implemented with the aim of meeting the needs of society, as well as with the efforts made by public agents to correct and expand public policies implemented in democratic environments.

In this sense, Hanberger (2011) linked the conceptualization of evaluation to the fulfillment of six functions (or variables): the improvement of policies and programs; internal accountability, reporting on process and performance; legitimation; the promotion of organizational learning; democratic and external accountability; and the fulfillment of symbolic / ritual functions.

Although the definitions for evaluation actions are many and broad, there is in common among them the idea that evaluations contribute decisively to generate the necessary knowledge to guide the decision-making process in public policies. Evaluations are therefore unique and valuable opportunities for organizational learning and policy improvement, based on four pillars - knowledge, learning, decision making, and improvement of the governmental environment. Hence, for the purposes of this analysis, the outcomes evaluation in public policies comprises the actions aimed at generating knowledge about the products and the results obtained by public policies, signaling if they are in line with its intentions and initial objectives. The evaluation actions are those that seek to provide information about the expected or unexpected consequences of the implementation of public policies, in order to maintain or correct the decisions adopted in their design or implementation phase, aiming to improve them based on the learning outcome (Anderson, 2008; Hanberger, 2011; Mark et al., 2000; Mark & Henry, 2004; Weiss, 1979, 1988b, 1999).

Douglass North’s (1991) contributions to this approach stem from his emphasis on the role played by institutions in societies. North (1991) expanded the concept of institutions, drawing attention to the possibilities for reducing uncertainties and to the limitations involved in institutionally mediated interactions, considering them as tools capable of connecting past, present and future. As stated by North (1991), institutions should be defined as formal and informal procedures, routines, norms and conventions immersed in the organizational structure of politics or economics, resulting from the competition between interest groups that interact with each other and with the State. Furthermore, North (1991) distinguished institutions from organizations, conceptualizing institutions as rules, and organizations as both the teams and the main agents of transformation in societies.

The role of national states in the development of innovative institutional changes and the repercussions on the political-institutional dynamics of governments was studied by North (1991), for whom such changes must be elaborated both on the basis of the relationships established between the structures that give them institutional support, as well as in the structural organizations that result from them. Skocpol (1985) and Evans (1993, 2008) analyses have added that organizational rules and factors influence political culture, bureaucracy, and state capabilities, identifying, among them, state capacity for policy evaluation.

Ostrom (2007) conceptualized rules as prescriptions, norms as internalized prescriptions, and strategies as plans, which tend to produce rules, norms, and expectations. In doing so, Ostrom (2007) associated rules, norms, and strategies to the contexts of implementation of public policies. Thus, Ostrom (2007) considered levels of operational analysis and collective, constitutional, and metaconstitutional choices, but her main focus was on the effect of rules on policy implementation.

Hofstede (2003), based on his studies on the cultural influences exerted by countries in organizational contexts, suggested that the term culture be considered, in organizational analyses, as a form of mental programming, noting that the collective programming of the mind distinguishes members from one group or category against another (Hofstede, 2003). Culture, in the broad sense, would be acquired and shared, coming from the social environment of individuals and groups. While recognizing the relevance of cultural factors to organizational structuring and decision making, McSweeney (2009) and Hall (2004) cautioned against the difficulties inherent in conceptualizing what are the cultural factors that surround an organization (Hall, 2004).

The term culture, in the broad sense of the word and adopted in the studies of social anthropology, has been considered as an attribute or internal quality to individuals or groups (Scott, 2004). Culture is also a collective phenomenon, which encompasses ways of thinking, feeling and behaving, while involving the sharing of these forms of expression among people living in the same social environment in which they are acquired (Hofstede, 2003).

To Weiss (1999) the degree of institutionality of a policy is associated with the cultural, historical, operational, budgetary and regulatory circumstances of public organizations. In this sense, the institutional path and the resolutions that precede the implementation of a policy impose strong determination on the conduct of the evaluation processes that will be established (path-dependence). Likewise, Weiss (1999) argues that evaluation actions participate in the institutional process by contributing to the incorporation of knowledge into new rules, new habits, and new organizational practices, in addition to concurring to changes in institutional culture and ideology.

In this perspective, the construct of institutionality adopted in this present analysis considers the term as the organizational assimilation of rules, both formal and informal, that derive from decisions and choices made during the period of formulation and implementation of public policies. The presence of these organizational regulation processes is relevant because it makes it difficult to reverse policies in the face of the expectations they generate, especially in situations where the political (institutional) cost of altering or suppressing policies is greater than the cost of maintaining them. In this sense, the clarity of rules, norms, and strategies is considered a factor associated with the stability and the possibility of obtaining results by public policies (Lane, 2014; North, 1991; Ostrom, 2007; Pierson, 2000; Streeck & Thelen, 2005; Weiss, 1999).

Therefore, the institutionality of a public policy can be evidenced by attributes such as (i) the insertion of politics in the organizations that participated in the process of institutional change in the area (Streeck & Thelen, 2005); (ii) the degree of formality of the policy (Anderson, 2008; Trosa, 2001); (iii) the level of debate between the organizations that participated in its formulation and implementation (Hall, 2004; Meny & Thoenig, 1992); and (iv) the degree of decentralization it presents, with regard to decision-making aspects and the execution of actions (Weiss, 1999).

3. THE EMPIRICAL GROUNDS OF DEBATE

In the literature, on the subject of the evaluation of public policies, there are several studies that point to the existence of associations between the institutionality of public policies and the chances that their results will be evaluated (Farrell & Héritier, 2004; Hall, 2004; Hanberger, 2011; Hofstede, 2003; Ingraham, 1987; Ingram, Schneider, & Deleon 2007; Kuhlmann & Shapira, 2009; Lane, 2014; Mahoney & Thelen, 2010; March, 2009; North, 1991; Ostrom, 2007; Pierson, 2000; Sabatier & Weible, 2007; Skocpol, 1985; Streeck & Thelen, 2005; Wandersman, 2014; Weiss, 1999, 1993, 1988a, 1988b, 1979).

The exploration of the studies that linked the outcome evaluation and the institutionality of public policies highlighted the set of analyses conducted by Weiss (1999, 1993, 1988a, 1988b, 1979), which deepened the relations established between the evaluation of public policies and the decisions made by policymakers or decision-makers. For Weiss (1999), the evaluation actions have the power to instruct the decisions of public policy managers, provided they are willing to consider the information derived from the evaluation process in policy making. As stated by Weiss (1999) the involvement of public policy managers in the conduct of evaluative actions is associated to the degree of comprehension they have about the power of instruction of evaluations for the formulation and implementation of public policies. Provided policy managers are committed to the so-called “good policies” and understand the power that evaluation actions have in order to maintain the adequacy of policies to the needs of the population, they tend to value them by using knowledge for the redesign of policies and the readjustment of its objectives and goals. Thus, for these public agents, the motivation to conduct evaluation processes would be based on the knowledge obtained in these processes, to obtain better and more adequate results, resulting from the implementation of public policies.

Based on the studies he carried out in the 1980s, Ingraham (1987) emphasized that the understanding of the processes of public agents’ choices regarding the implementation of evaluative processes requires conducting analyses that consider the variables present in the context of policy formulation.

In this sense, Ingraham (1987) focused on the complexity that involves the decisions implied in the design process of a public policy, which includes evaluation actions, and emphasized the relevance of the choices made by public agents in the formulation of specific indicators for the organization of evaluation actions. It is worth mentioning that there are situations in which difficulties to show the results of public policies may not be due to the lack of results in regard to the objectives proposed by the policies. In some cases, the evaluation of results is hampered by gaps arising from inconsistent policy designs (Ingraham, 1987; Hanberger, 2011; Howlett, 2014).

Among the reasons identified by Weiss (1999) to explain the low frequency for conducting public policy evaluation actions, the instances of institutional disconnection between the formulation process and the policy evaluation process deserve special mention. In contrast, the decentralized deployment of policies, competitive political systems, and democratic systems were identified as facilitators of the institutionalization of evaluation actions. Moreover, Weiss (1999) pointed out that, despite the relevance of evaluation processes to pressure groups, members of the legislature and legislators, party leaders, taxpayers, groups potentially benefited by policies, researchers, among others, the public agents or policymakers constitute the group that is most directly impacted, while also impacting on the results of the implementation of the public policies in which they are involved.

Pierson (2000) has associated the occurrence of specific events (particularly those that are rule-makers) to institutionalized organizational practices, in which process and outcomes evaluations can be included. Pierson’s analyses (2000) called attention to conjunctures in which great political changes occur, which are characterized by interfering in the positions assumed by the actors in the political game, changing the rules and instruments of institutional power.

On the one hand, the theoretical matrix of Pierson (2000) has contributed to explain how the chain of previous decisions explains later trajectories assumed in the implementation and results of public policies, configuring them as path dependent. On the other hand, when analyzing how government actions complement the decisions adopted in periods of great changes, his analyses addressed other theoretical models focused on the institutionality of public policies, also present in the studies carried out by Mahoney and Thelen (2010), Streeck and Thelen (2005) and North (1991). Notwithstanding the specificities of each of these models, they all affirmed the importance assumed by the changes in the rules as a form of construction of viability, with repercussions on the implementation and results of the public policies.

Some studies have clarified the situations in which variables of a cultural nature would be related to the absence of evaluative actions in processes of implementation of public policies (McSweeney, 2002, 2009; Ostrom, 2007; Scott, 2004; Trompenaars, 1994). Based on the link between cultural resistance, learning, and organizational changes, Hofstede (1997) found that the behavioral, feeling and thinking patterns of certain policymakers are likely to provoke interference in decisions regarding the formulation of public policies. For Hofstede (2003) these standards interfere in the design of public policies and in the way in which their results will be evaluated.

Kuhlmann and Shapira (2009) and Farrell and Héritier (2004) found associations between the institutionality of public policies and the results obtained by them in industrialized countries. Kuhlmann and Shapira (2009)analyzed the relationships between governance styles, policy, and results on innovation and found evidence that state regulation influences both the form of evolution and the results produced by innovation systems. Hanberger (2011) associated the assembly of evaluation systems with good governance, and with the political, institutional and procedural arrangements related to the internal and external environment of public policies. He emphasized that policy design and managerial support are critical factors for the implementation and operation of evaluation systems.

In the same analytical trend, Mahoney and Thelen (2010) and Streeck and Thelen (2005) emphasized the importance of institutionality for the recognition and outcome evaluation in public policy implementation processes, highlighting both the weight of institutional characteristics of policies and their interactions with the public agents, in processes of institutional change. Streeck and Thelen (2005) highlighted the relevance of the rules and standards established for the recognition of the degree of institutionality reached in public policymaking processes, and emphasized the importance of political contexts, of institutional characteristics and their interaction with public agents in processes of institutional change.

The emphasis of the institutionality of the policy for the conduction of outcomes evaluations was developed in other studies carried out by authors who are affiliated to the institutionalist side, among which the analyses carried out by March (2009) and Sabatier and Weible (2007).

The model of analysis conjectured by Sabatier and Weible (2007), the Advocacy Coalitions Framework (ACF), predicted that the institutionality of a policy leads to situations of stability, this being the case of public policies that have been maintained for more than a decade. Sabatier and Weible (2007) associated stable public policies with normatively constructed results evaluation models, whose objectives included: (i) the link between public policy objectives and the values they sought to build; and (ii) the focus on the possibilities for reformulation and revision as a result of appraisal-based learning.

In the Advocacy Coalitions Framework (Sabatier & Weible, 2007), the policy-making process is viewed from a belief system and understood as the mode of interpretation of problems, which is both capable of generating cohesion and mistrust among process participants. Sabatier and Weible (2007) distinguished policies external factors as stable (those which remain for more than a decade) and dynamic ones (those which vary in shorter periods). For Sabatier and Weible (2007), the temporal factor is considered as a relevant variable for the identification of the stability of public policies and should be considered both for the proposal of their evaluation indicators and in their evaluation design.

Sabatier and Weible (2007) proposed a policy analysis’ model that contemplated the decisions adopted in the public policy formulation processes and, therefore, also contributed to the construction of approaches aimed at outcome evaluation. Those authors strongly related the evaluation of results in public policies to four factors: (i) the set of decisions that were manifested during the formulation of the policy; (ii) the formulators’ ability to clearly state the objectives of the policy; (iii) the factors related to the products and the results of the policy; and, (iv) the possibilities of readjustments contemplated in the policy design, provoked by the knowledge obtained in the course of the implementation process.

Within the approach proposed by Sabatier and Weible (2007), results evaluation processes should be defined through the elaboration of proposals built with strong normative content. According to those authors, evaluation actions should contemplate and express the rules’ design, whose comprehensiveness conveys both the link between the values and the objectives of the policy, as well as possibilities of adjustments and course changes based on knowledge and learning obtained from the evaluation process.

In a subsequent work than that of Sabatier and Weible (2007), March (2009) presented an institutional learning model for the evaluation of public policies whose starting point consists of inquiries about what happened, why it happened and whether what happened was satisfactory or unsatisfactory (March, 2009). March’s work has greatly valued the historical course of organizations and societies, recommending that the answers to the questions posed at the initial moment of evaluation processes should incorporate historical events, which are sometimes not considered as objects of observation. In addition to these historical events, March suggests that be considered, among others, both the concrete events and the events that have been observed, as well as the interpretations of policymakers about them. As stated by March (2009) the institutional learning processes that guide the evaluation of results use institutional analyses, as well as selections, bargains, and imitations. Thus, these institutional factors began to influence and, therefore, to assume relevance for analyses that consider the relation between learning and the evaluation of results.

Additionally, March (2009) drew attention to the fact that institutional analyses involving the participation of different individuals should consider that the assimilation of learning in these contexts occurs differently. Thereby, even if the participants in the implementation of public policy have experienced the same historical processes, their experiences are affected by factors such as recalls, interpretations, and evaluations made by themselves (March, 2009).

Situations in which public policy evaluation processes produce contradictory results, indicating success or failure according to the evaluator’s situation, were also analyzed by March (2009). The author identified a tendency to self-congratulation by the evaluators, which would lead to adjustments both in the aspiration levels and in their definitions of what should and should not be valued in evaluation processes. In this sense, variables such as experience and the elaboration of value judgments by the evaluators are important variables to explain the results of evaluations that consider the gap between the initial expectations and the results obtained.

In these terms, Anderson (2008) pointed out that the lack of information about the results obtained from a policy would act as an obstacle to the identification of factors that contribute to its success or failure. These obstacles would be expressed as difficulties in making decisions about the necessary adjustments in regard to the definition and reformulation of the policies, in order to continue, modify, strengthen or even end their implementation.

Wandersman (2014) addresses perspectives developed by Anderson (2008) and Weiss (1999). Wandersman (2014) proposed that evaluations should be empowered and that the evaluative capacity be considered organizational constructs, involving such factors as: (i) the assessment of needs and resources; (ii) the definition of the desired objectives and results; (iii) the selection of evidence-based practices for policies; (iv) the necessary adjustments; (v) the implementation and process evaluations; and, (vi) sustainability.

For Wandersman (2014) evaluation processes should be viewed from a perspective of continuous improvement of public organizations, in which information is organized and applied both for the self-assessment of policymakers and for the implementation of quality public policies. He suggests that the ability to plan in a systematic way is a function of the ability to use the knowledge and evaluation tools, which are, ultimately associated with the assessment capabilities of public policy managers.

An effort to synthesize the main contributions of the target authors, the relationships observed between the selected variables and the deriving conclusions are seen in Box 1.

BOX 1
RELATIONSHIPS BETWEEN THE INSTITUTIONALITY OF PUBLIC POLICIES AND EVALUATION PROCESSES

4. CONCLUSION

As a conclusion, it is worth recalling the two questions that guided the study, namely: (i) which variables can be used to evidence the institutionality of public policies? (ii) are more institutionalized public policies more likely to have their results evaluated?

Regarding the first question, several of the studied authors identified variables capable of indicating the degree of institutionality of public policies, pointing to associations that, although remaining fluid, can be used in future studies on the subject. Viewed in this way, the studies conducted by Howlett (2014), Cousins et al. (2014), Hanberger (2011), Anderson (2008), Mark et al. (2000), Meny and Thoenig (1992), Ingraham (1987) and Weiss (1999, 1988b, 1979) revealed that public policies whose designs contains processes aimed at knowledge management, learning, improvement of structures and organizational processes have greater institutionality.

For authors like Mahoney and Thelen (2010), Streeck and Thelen (2005), Evans (1993, 2008), Pierson (2000), Weiss (1999), Meny and Thoenig (1992) and Skocpol (1985) public policies whose organizational rules and factors influence political culture, bureaucracy, and state capacity for action, among which the ability to evaluate results, can be considered more institutionalized than those policies in which these attributes are not identified. It is worth mentioning that Weiss (1999) adds to these associations the political formality, the high level of debate between the organizations participating in the formulation and implementation of the policy and the decentralization in the decision making and execution aspects, emphasizing their relevance to the definition of the institutionalized aspect of public policies.

The second aspect that guided this study concerns the associations between the institutionality of public policies and outcomes evaluations, that is, whether and how the institutionality of a public policy favors or not the evaluation of its results. In this sense, the analyzed studies identified variables that, when present, indicate that the institutionality of public policy can be positively or negatively associated with the processes of outcomes evaluation. Hence, the positive associations between public policies with a high degree of institutionality and the conduction of results evaluations were grouped, for didactic purposes, into three sets of variables: (i) variables that reflect the characteristics of the decision-making process and the policy implementation context; (ii) identifiable variables in the policy design; and (iii) analyzable variables based on the behavior of policymakers.

The first set of variables reflects the characteristics of the decision-making process and the policy implementation context, evoking the way in which the policy objectives are linked to the values they seek to construct. In this set, are relevant the variables that express the link between the objectives of the policy, the clarity of its purposes and the possibility of making adjustments in the policy, based on the lessons learned, (Hanberger, 2011; Sabatier & Weible, 2007), as well as the variables related to the decentralized implementation of policy and the adoption of democratic and participatory decision-making processes (Trosa, 2001; Weiss, 1999). The political-institutional contexts that include values such as visibility, transparency, accountability to citizens and the possibility of correcting the evaluated policy were also identified as favorable to outcome evaluations (Faria, 2012; Mark et al., 2000; Meny & Thoenig, 1992; Trosa, 2001). These contexts value the support and the way people, either individually or collectively, give meaning to the policies implemented to meet human needs (Meny & Thoenig, 1992; Trosa, 2001).

The second set of variables values not only the policy design but also the organizational commitment towards learning, political-institutional improvement, adaptation and the possibility of reformulation and revision of the policy, which increases the chances of policies to have its results evaluated (Cousins & Leithwood, 1986; Cousins et al., 2014; Hanberger, 2011; March, 2009; Mark & Henry, 2004; Meny & Thoenig, 1992; Pressman & Wildavsky, 1984). According to studies by March (2009), Anderson (2008), Ostrom (2007), Streeck and Thelen (2005) and Trosa (2001), when policy design covers the adoption of clear rules and information systems, as well as of norms and strategies capable of influencing the political culture, bureaucracy, and state capabilities, the likelihood of evaluating the results of the policies are increased.

The third set consists of the variables whose analysis is based mainly on the behavior of policymakers. Those analyses are based on the recognition of the managers of the policy on the merit and the value of the evaluation of results for the organizational and programmatic improvement, for the supervision and the development of the knowledge about the policy. In other words, when the concern for social betterment or the opportunity to redesign politics is manifested and assumed by policymakers, in order to adapt it to the proposed objectives, with the aim of obtaining better results, the chances of evaluating results of public policies increases (Hanberger, 2011; Mark et al., 2000; Mark & Henry, 2004; Meny & Thoenig, 1992; Trosa, 2001; Weiss, 1988a).

The negatively established associations between the institutionality of public policies and the conduction of outcomes evaluations were explained by factors such as the degree of informality and cultural resistance. The high degree of informality and the low level of debate in the process of formulating and implementing a public policy tend not to make way to the knowledge offered by the outcomes of evaluation actions, given that in these contexts each actor involved in the process tends to act with its own level of personal knowledge, not valuing the delineation offered by the actions and evaluation systems (Weiss, 1999). Cultural resistance and frequent changes in the organizational environment that alter thinking, feeling and behavior patterns of policymakers have been negatively associated with the processes of evaluation, due to the instability that they generate in the decision making processes and in the trajectory of policy implementation (Hofstede, 2003; McSweeney, 2009; Meny & Thoenig, 1992; Ostrom, 2007; Scott, 2004; Trompenaars, 1994).

In summary, the survey carried out on the evaluation of public policy results identified that the understanding of the path taken by public policies may benefit from approaches based on explanatory models that are guided by the neoinstitutionalist perspective. In this regard, it should be pointed out that, for the purposes of this essay, the term institutionality derives from an expanded concept of institutions, such as proposed by North (1991) and Ostrom (2007), which encompasses many types of entities, and includes the organizations and rules used to structure interaction models with and within organizations.

In these terms, it is recommended that analyses on the subject of outcomes evaluation consider the influences exerted by variables related to institutional trajectories, actors and organizational contexts both at the moment of the formulation of public policies and during the phase of their implementation, since all these influences act in the situations in which decisions are made regarding the evaluation process.

The work carried out on the subject of outcomes evaluation revealed that the institutionality of policies deserves to remain an object of attention in studies conducted on evaluation in public policies. In addition, the study showed that the debate on the evaluation of public policies can be strengthened by analytical strategies and the conduction of empirical studies that consider the decisions adopted by the public policy makers, as well as the rules, norms and strategies that define the political-institutional scene in which public policies are implemented.

REFERENCES

• Anderson, J. (2008). Public Policymaking: an Introduction (7 ed.). Boston, Massachusetts: Houghton-Mifflin.
• Batista, M., & Domingos, A. (2017) Mais que boas intenções: técnicas quantitativas e qualitativas na avaliação de impacto de políticas públicas. Revista Brasileira de Ciências Sociais, 32(94), 1-24. Retrieved from http://dx.doi.org/10.17666/329414/2017
  » http://dx.doi.org/10.17666/329414/2017
• Bourgeois, I., & Cousins, J. B. (2013). Understanding dimensions of organizational evaluation capacity. American Journal of Evaluation, 34(3), 299-319.
• Bovaird, T. (2014). Attributing outcomes to social policy interventions - ‘gold standard’ or ‘fool’s god’ in public policy and management?. Social Policy and Administration, 48(1), 1-23.
• Cousins, J. B., & Leithwood, K. A. (1986). Current empirical research on evaluation utilization. Review of Educational Research, 56(3), 331-364.
• Cousins, J. B., Goh, S. C., Elliot, C. J., & Bourgeois, I. (2014). Framing the capacity to do and use evaluation. Special issue: organizational capacity to do and use evaluation. New Directions for Evaluation, 141, 7-23.
• Crumpton, C. D., Medeiros, J. J., Ferreira, V. da R. S., Sousa, M. de M., & Najberg, E. (2016). Evaluation of public policies in Brazil and the United States: a research analysis in the last 10 years. Revista de Administração Pública, 50(6), 981-1001.
• Cunill-Grau, N., & Ospina, S. M. (2012). Performance measurement and evaluation systems: institutionalizing accountability for governmental results in Latin America. New Directions for Evaluation, 134, 77-91.
• Evans, P. (1993). O Estado como problema e solução. Lua Nova: Revista de Cultura e Política, 28-29, 107-157.
• Evans, P. (2008). In search of the 21st century developmental state Sussex, UK: Centre for Global Political Economy at the University of Sussex. (Working Paper Nº 4, p. 22). Retrieved from https://www.sussex.ac.uk/webteam/gateway/file.php?name=cgpe-wp04-peter-evans.pdf&site=359
  » https://www.sussex.ac.uk/webteam/gateway/file.php?name=cgpe-wp04-peter-evans.pdf&site=359
• Farrell, H., & Héritier, A. (2004). Interorganizational negociation and interorganizational power in shared decision making. Early agreements under codecision and their impact on the European parliament and council. Comparative Political Studies, 37(10), 1184-1212.
• Faria, C. A. P. (2012). Implementação de Políticas Públicas Teoria e Prática Belo Horizonte, MG: Editora PUC de Minas Gerais.
• Hall, R. H. (2004). Ambientes organizacionais e relacionamentos interorganizacionais. In: Hall, R. H. Organizações: estruturas, processos e resultados (8 ed., pp. 191-223). São Paulo, SP: Pearson Prentice Hall.
• Hanberger, A. (2011). The real functions of evaluation and response systems. Evaluation, 17(4), 327-349.
• Hansen, H., Klejnstrup, N. R. & Andersen, O. W. (2013). A comparison of model-based and design-based impact evaluations in developing countries. American Journal of Evaluation, 34(3), 320-338.
• Hill, M., & Hupe, P. (2005). Implementing public policy (2 ed.). Thousand Oaks, California: Sage Publications Inc.
• Hofstede, G. (2003). Culturas e Organizações Compreender a nossa programação mental. Lisboa, Portugal: Edições Sílabo.
• Howlett, M. (2014). From the ‘old’ and the ‘new’ policy design: design thinking beyond markets and collaborative governance. Policy Sci, 47(3), 187-207.
• Ingraham, P. (1987). Toward more systematic consideration of policy design. Policy Studies Journal, 15(4), 611-628.
• Ingram, H., Schneider, A. L., & Deleon, P. (2007). Social construction and policy design. In: P. A. Sabatier(Ed.), Theories of the Policy Process (7 ed., pp. 93-128). Boulder, Colorado: Westview Press.
• Kuhlmann, S., & Shapira, P. (2009). How is innovation influenced by science and technology policy governance? Transatlantic comparisons. In J. Hage & J. Meeus(Eds.), Innovation, Science, and Institutional Change: A Research Handbook. (2 ed., pp. 232-253). New York, NY: Oxford University Press.
• Labin, S. N. (2014). Developing common measures in evaluation capacity building: an iterative science and practice process. American Journal of Evaluation, 35(1), 107-115.
• Lane, J. E. (2014). Institutionality: “institution” and “institutions matter”. Open Journal of Polítical Science, 4(1), 23-30.
• March, J. G. (2009). Como as decisões realmente acontecem: princípios de tomada de decisões São Paulo, SP: Editora Leopardo.
• Mahoney, J., & Thelen, K. (2010). Explaining institutional change. Ambiguity, agency and power Cambridge, UK: Cambridge University Press.
• Mark, M. M., & Henry, G. T. (2004). The mechanisms and outcomes of evaluation influence. Evaluation, 10(1), 35-57.
• Mark, M. M.; Henry, G. T. & Julnes, G. (2000). Evaluation: an integrated framework for understanding, guiding, and improving policies and programs San Francisco, CA: Jossey-Bass.
• Mastenbroek, E., Voorst, S. V., & Meuwese, A. (2015). Closing the regulatory cycle? A meta evaluation of ex-post legislative evaluations by the european commision. Journal of European Public Policy, 23(6), 1329-1348. Retrieved fromhttp://www.tandfonline.com/doi/abs/10.1080/13501 763.2015.1076874?journalCode=rjpp20
  » http://www.tandfonline.com/doi/abs/10.1080/13501 763.2015.1076874?journalCode=rjpp20
• Mc Sweeney, B. (2002). Hofstede’s model of national cultural differences and their consequences. Human Relations, 55(1), 89-118.
• Mc Sweeney, B. (2009). Dynamic diversity: variety and variation within countries. Organization Studies, 30(9), 933-957.
• Meny, I., & Thoenig, J. C. (1992). Las políticas públicas, Barcelona, Espanha: Editorial Ariel.
• North, D. C. (1991). Institutions. The Journal of Economic Perspectives, l(5), 97-112.
• Oliver, K., Innvar, S., Lorenc, T., Woodman, J., & Thomas, J. (2014). A systematic review of barriers to and facilitators of the use of evidence by policymakers. Health Services Research, 14(2). Retrieved fromhttp://bmchealthservres.biomedcentral.com/articles/10.1186/1472-6963-14-2
  » http://bmchealthservres.biomedcentral.com/articles/10.1186/1472-6963-14-2
• Oliver, K., Lorenc, T. E., & Tinkler, J. (2019). Evaluating unintended consequences: new insights into solving practical, ethical and political challenges of evaluation. Evaluation, 1-15. Retrieved fromhttps://doi.org/10.1177/1356389019850847
  » https://doi.org/10.1177/1356389019850847
• Oliveira, L. R., & Passador, C. S. (2019) Theoretical essay on public policy evaluations. Cadernos EBAPE.BR, 17(2), 324-337.
• Oliveira, V., & Pinho, P. (2011). Bridging the gap between planning evaluation and programme evaluation: the contribution of the PPR methodology. Evaluation, 17(3), 293-307.
• Ostrom, E. (2007). Institutional rational choice: an assessment of the institutional analysis and development framework. In: P. A. Sabatier (Ed.), Theories of the policy process, (7 ed., pp. 21-64). Boulder, Colorado: Westview Press .
• Patton, M. Q. (1988). Reports on topic areas. The evaluator’s responsibility for utilization. Evaluation Practice, 9(2), 5-24.
• Patton, M. Q. (1999). Organizational development and evaluation. The Canadian Journal of Program Evaluation, Special Issue, 93-113.
• Pierson, P. (2000). Increasing returns, path dependence and the study of politics. American Political Science Review, 94(2), 251-267.
• Pressman, J. L. & Wildavsky, A. (1984). Implementation (3 ed.). Berkeley: University of California Press.
• Ramos, M. P. & Schabbach, L. M. (2012). O estado da arte da avaliação de políticas públicas: conceituação e exemplos de avaliação no Brasil. Revista de Administração Pública, 46(5), 1271-294.
• Richardson, L. (2013). Putting the research boot on the policymakers’s foot: can participatory approaches change the relationship between policymakers and evaluation? Social Policy and Administration, 47(4), 483-500.
• Sabatier, P. A., & Weible, C. (2007). The advocacy coalition framework: innovations and clarifications. In: P. A. Sabatier (Ed.), Theories of the policy process (7 ed., pp. 189-220). Boulder, Colorado: Westview Press .
• Scott, W. R. (1998). Organizations. rational, natural, and open systems (4 ed.). New Jersey, USA: Prentice-Hall Inc.
• Scott, W. R. (2004). Institutional theory: contributing to a theoretical research program. In: Smith, K. G.; Hitt, M. A. (eds.). Great minds in management: the process of theory development Oxford UK: Oxford University Press.
• Serapioni, M. (2016). Conceitos e métodos para a avaliação de programas sociais e políticas públicas. Sociologia: Revista da Faculdade de Letras da Universidade do Porto, 31, 59-80. Retrieved from http://www.scielo.mec.pt/scielo.php?script=sci_arttext&pid=S0872-34192016000100004&lng=pt&nrm=iso
  » http://www.scielo.mec.pt/scielo.php?script=sci_arttext&pid=S0872-34192016000100004&lng=pt&nrm=iso
• Skocpol, T. (1985). Bringing the state back in: strategies of analysis in current research. In P. B. Evans; D. Rueschemeyer, & T. Skocpol (Eds.). Bringing the state back in (pp. 3-37). Cambridge, UK: Cambridge University Press .
• Streeck, W., & Thelen, K. (2005). Beyond continuity: institucional change and advanced political economies New York, NY: Oxford University Press .
• Suarez-Balcazar, Y., & Taylor-Ritzler, T. (2013). Moving from science to practice in evaluation capacity building. American Journal of Evaluation, 35(1), 95-99.
• Trompenaars, F. (1994). Nas ondas da Cultura Como entender a diversidade cultural nos negócios São Paulo, SP: Educator.
• Trosa, S. (2001). Gestão pública por resultados Quando o estado se compromete Rio de Janeiro, RJ: Revan; Brasília, DF: Enap.
• Vedung, E. (1997). Public policy and program evaluation New Bruswick, NJ: Transaction Publishers.
• Wandersman, A. (2014). Getting to outcomes: an evalutation capacity building example of rationale, science, and practice. American Journal of Evaluation, 35(1), 100-106.
• Weiss, C. H. (1979). The many meanings of research utilization. Public Administration Review, 5(3), 426-431.
• Weiss, C. H. (1988a). If program decisions hinged only on information: a response to Patton. American Journal of Evaluation, 9(3), 15-28.
• Weiss, C. H. (1988b). Reports on topic areas. Evaluations for decisions: is anybody there? Does anybody care? American Journal of Evaluation, 9(1), 5-19.
• Weiss, C. H. (1993). Where politics and evaluation research meets. Evaluation Practice, 14(1), 93-106.
• Weiss, C. H. (1998). Have we learned anything new about the use of evaluation? American Journal of Evaluation, 19(1), 21-33.
• Weiss, C. H. (1999). Speeches and addresses. The interface between evaluation and public policy. Evaluation, 5(4), 468-486.
• Weiss, C. H. (2010). Perspectivas de implementação: status e reconsideração. In B. G. Peters, J. Pierre (Orgs.), Administração Pública (Coletânea, pp. 209-228). São Paulo, SP: Editora UNESP; Brasília, DF: Enap.
• Winter, S. (1990). Integrating implementation research. In D. J. Palumbo, & D. Calista (Orgs.), Implementation and the policy process: opening up the black box(6 ed., pp. 19-50). New York, NY: Greenwood Press.

• [Original version]

Publication Dates

• Publication in this collection
  26 June 2020
• Date of issue
  May-Jun 2020

History

• Received
  22 July 2019
• Accepted
  26 Sept 2019