«Abstract: ROI (Return on Investment) is an increasing consideration of government and other funders with respect to activities that they support, ...»
A Holistic Model Of Outcome Evaluation For Arts Engagement
PhD candidate, Deakin University, Melbourne, Australia.
Paper presented at the Australasian Evaluation Society International
Conference, Brisbane, Australia, 2 - 6 September 2013.
Abstract: ROI (Return on Investment) is an increasing consideration of government
and other funders with respect to activities that they support, including the arts. This
approach is used ostensibly to ensure most prudent use of limited public resources.
However, a challenge with this approach in arts contexts is that, as ROI is generally only considered in economic terms, much of the benefit of arts engagement can be missed. A similar under-assessment is made by evaluation processes that only examine social outcomes of arts engagement.
This paper proposes a model of evaluation for outcomes of arts engagement that takes a holistic approach. Categories for outcome evaluation are drawn from two models of community sustainability; Hawkes’ (2001) four pillars of sustainability and Ife’s (2002) six dimensions of community development. This comprehensive approach eliminates the need for the problematic intrinsic and instrumental classifications of outcomes of arts engagement. The model is illustrated by case examples from the author’s research into participatory arts initiatives in Timor-Leste.
A pictorial representation of outcomes of arts engagement offers a quantitative perspective on qualitative data. In this, the model addresses a further challenge for qualitative researchers who need to present easily digestible results from large amounts of data.
The challenges in evaluating impact of the arts Evaluation of participatory arts is a burgeoning practice, with evaluation taking place more frequently across the spectrum from academic research to project related assessments, driven by funder requirements for accountability and increasing interest by program hosts in effective practice. However, evaluation of arts participation typically has a number of limitations as a result of practical and conceptual challenges. The first is that outcomes are often reported from the perspective of limited sub-section of stakeholders. Organisers’ perspectives have traditionally provided a major source of data for all types of arts engagement (McQueen-Thomson, James & Ziguras, 2004). These are limited for the obvious reasons of bias (Etherton & Prenkti, 2007).
In initiatives that focus on participation, the next most frequently considered group in outcome evaluation are active participants. In arts initiatives that focus on receptive participation, such as theatre or gallery attendance, strategies for considering impact on audiences are growing (Radbourne, Johanson & Glow, 2010; Brown & Ratzkin, 2012). However, fewer evaluation approaches apply integrated methods, where perspectives of a range of stakeholders, which might include participants, audience members, the wider community, arts leaders, other staff, funders and policy makers, are considered.
In other fields, awareness of the different perspectives and priorities of stakeholders has led to growth of methodologies that accord value to different worldviews. The methodology Most Significant Change (MSC) is used to provide a framework for systematic consideration of input from different stakeholders, from program beneficiaries through to program managers and funders (Davies & Dart, 2005). MSC is also being used in evaluation of participatory arts programs (Johnson & Stanley, 2007; Laidlaw Foundation, 2010).
The next recurrent challenge for outcome evaluation of arts initiatives is the issue of what to measure and how to measure it (South, 2004). Evaluation strategies about arts participation have traditionally focussed only on outputs, that is, the amount of opportunity provided (for example, as measured by numbers and diversity of participants), rather than outcomes, in terms of the contribution to desired goals (Matarasso, 1996; Blomkamp, 2011). This challenge is being addressed in other fields as well, with growing concern to prioritise outcome over output evident in international development (Morra-Imas & Rist, 2009) and local government (West & Cox, 2009).
However, when outcomes are emphasised in arts evaluation, a further challenge is encountered; the complexities of measuring value and outcomes of the arts because of their perceived intangible nature. These difficulties are evidenced in the persistent tensions about the intrinsic versus instrumental valuing of the arts, where intrinsic value is considered to pertain to the quality of the arts experience itself and instrumental seen as the contribution of the arts to other goals, such as social or economic progress (McCarthy, Ondaatje, Zakaras & Brooks, 2004;
Throsby, 2001; Holden, 2006).
McCarthy et al (2004) argue that intrinsic effects of the arts preclude the need for any other justification. These include immediate benefits of pleasure and captivation;
growth in individual capacities such as enhanced empathy for other people and cultures; and benefits that accrue largely to the public, including the social bonds created and expression of community identity (2004, p. 56). Throsby proposes a different perspective of intrinsic value, identifying six categories of aesthetic, spiritual, social, historical, symbolic and authenticity value (2001, p. 28-29).
An opposing view of the benefits of arts participation is instrumentalist, in which value is measured by agendas considered outside the arts, such as economic or social goals. O’Brien’s (2010) comprehensive literature review concluded that economic value is the only useful frame for evaluation of the arts as it is the major decision point for governments. A different instrumental view recognises the value of the arts for its contribution to social outcomes, including health, wellbeing and social inclusion (McQueen-Thomson et al, 2004; Kelaher et al, 2007). Another instrumental agenda examined in more recent research is the contribution of arts participation to civic engagement goals (Stern & Seifert, 2009; Mulligan and Smith, 2010).
Cultural analyst Jon Hawkes responds to this dilemma by positing that intrinsic and instrumental categorisations are only necessary because the appropriate instrumental measures have not been considered (Hawkes, 2010). He proposes that all value of arts participation is instrumental and can be categorised within one of the four dimensions of social equity, economic viability, environmental responsibility and cultural vitality. While Hawkes does not offer an evaluation framework, he does argue that arts and all other areas of public policy and investment should be subject to evaluation against those four dimensions because they are equally important aspects of a meaningful and sustainable human existence (Hawkes, 2001).
While measurement of social and economic impacts of arts participation is not new, Hawkes does pose two categories that are conceptually more recent: environmental sustainability and cultural vitality. Assessments of environmental impact are being undertaken with increasing frequency in many areas of public endeavour, although this consideration is not yet commonplace within arts evaluation. Hawkes’ most significant contribution is the addition of a cultural vitality dimension, which he posits as an essential aspect of any holistic framework for planning or evaluation. At the time of his writing in 2001, this was an infrequently considered dimension, with culture mostly not considered or subsumed under other headings, such as social.
The cultural dimension is still infrequently included in measurement frameworks because of the perception that its intangible nature renders it immeasurable (McGillivray, 2009).
A related problem is the common use of ‘arts’ and ‘culture’ as interchangeable terms.
This makes measurement of the impact of arts on culture tautological, and therefore problematic. However, if the arts are seen as one dimension of a much broader concept of culture, then it is possible to consider the cultural impact of arts initiatives.
In so doing, the troubling intrinsic/instrumental dichotomy can be addressed.
Arguments for the intrinsic value of the arts (or any other concept) are particularly risky because of their circular nature. A proposition that a concept or activity is valuable by its very nature, that is, that arts participation is valuable because it is arts, must be seen as flawed. A stronger argument can only be made against a set of shared ideas about what is valuable. What do we value, and therefore what do we need to measure, about arts participation, or any other activity?
If Hawkes’ framework is applied, almost all of the concepts that Throsby considered as intrinsic contributions of the arts can be reframed as instrumental. His aesthetic, spiritual, historical, symbolic and authenticity values can be considered within the cultural dimension. Social value can be considered within the dimension of social equity. Aspects that McCarthy et al consider as intrinsic such as “enhanced empathy for other people and cultures”, and “social bonds” (2004, p. 56), fit those dimensions well.
McCarthy’s “pleasure and captivation”, “expression of common values and community identity” and “understanding of the world” are not so easily categorized within this four pillar model. However, a model related to Hawkes’, but offering two additional dimensions offers a solution. Ife’s model of integrated community development has six categories: the same four as Hawkes, of social, cultural, environmental and economic, but also two additional categories of civic engagement and personal-spiritual well-being (Ife, 2002). If these six categories are used, they allow for categorisation of “pleasure and captivation” within the personal-wellbeing dimension, and “expression of community identity” and “understanding of the world” within the civic engagement dimension. By offering a comprehensive framework for categorising outcomes of arts participation, these models eliminate the need for the problematic conception of intrinsic value. They also connect thinking about arts participation with emerging frameworks for human progress.
Frameworks for human wellbeing and development are increasingly holistic, considering all aspects of human experience and the natural world as interconnected. Along with Hawkes and Ife’s ideas, such approaches include Clammer’s “vision of integrated development in which social, economic, cultural, political, spiritual and environmental elements are holistically related” (Clammer, 2012, p. 53).
James’ Circles of Sustainability acknowledges culture, politics, economy and ecology as interlinked dimensions within a larger framework of social sustainability (Global Compact Cities Program, 2013).
Arts evaluations are not yet indicating significant influence from these broader conceptual approaches. One significant recent exception is Cooper, Bahn and Giles (2012) who considered dimensions of economic, social, artistic and personal wellbeing in their evaluation of an indigenous arts centre.
In addition to all of the aforementioned challenges of arts evaluation, further limitations are that evaluations employing qualitative methodologies tend to be limited in consideration of direction and degree of change. Evaluation frameworks like those proposed by Morra-Imas and Rist (2009) recommend consideration of the full range of directions: positive and negative, direct and indirect, intended or unintended change. However these different directions are infrequently considered in arts outcome evaluations with assumptions of positive change prevalent (Rosenberg, 2008; Galloway, 2009). The degree of change is also infrequently specified, which leads to difficulties in determining whether resources were used wisely.
Considerations about return on investment (ROI) are increasingly included in policy and decision-making to provide assessment of effective use of resources (Farris, Bendle, Pfeifer and Reibstein, 2010). While traditional uses of ROI focus on economic return, this is being challenged by wider conceptions of social return on investment (Goodspeed, Lawlor, Neitzert & Nicholls, 2009). Again, however, integrated approaches are as yet not applied. Few arts evaluations consider amount of change proportionate to investment across a range of dimensions.
One final challenge for arts evaluation is shared with other fields in which qualitative methodologies are most frequently used. The complex findings from qualitative research can be difficult to express and share in ways that are accessible to those who make decisions, who need concise and compelling information founded on robust data. Jupp, Ali and Barahona’s (2009) model for quantifying qualitative data offers an interesting response to this dilemma, as it was developed to measure empowerment, a concept like the intrinsic value of the arts is considered problematic because of its amorphous nature. Some of the strategies Jupp et al suggest such as development of project indicators by project participants and use of symbols to reflect direction and degree of change, may offer a way forward for arts research as well.
Presenting the model To address the challenges discussed above, a holistic evaluation model is presented.
This offers a means of considering different perspectives of change (who perceived and experienced the change); dimensions of change (what kind of change occurred) and degree of change (how much change occurred).
Taking this approach, data is first categorised according to the perspectives of different stakeholders. This strategy is informed by Most Significant Change methodology (Dart & Davies, 2003), which acknowledges that an initiative might also impact on these stakeholders in different ways and these different experiences need to be considered in an evaluation. Figure 1 below offers a pictorial representation of the different types of stakeholders and their proportionate significance. This indicates that a project might primarily be expected to impact participants, but also audience members, artistic leaders, the host organisation, funders and the wider community, in lesser degrees, and each of these groups might perceive outcomes differently. These circles can be adjusted in response to the particular circumstances of any initiative or evaluation focus.