Two days (27-28 April), 16 learning sessions, lots of networking! Lots of international development content which took a bit of navigating for evaluators working in a UK/EU context (while still picking up some very helpful info from globally-based colleagues!). Some initial reflections below, along with a full list of sessions attended. I’d expect that all of the presenters would be happy to field any requests for further information, although I’d also certainly be happy for any follow-up discussions, particularly on the Voluntary Evaluator Peer Review pilot/process which I’ve participated in and would highly recommend.
Choosing methods. There was a sense of some RCT pushback among attendees and presenters, not least from the newly launched CECAN which will be looking into alternative approaches using realist context, simulated RCTs / policy modelling, drawing on participatory expertise and ongoing monitoring / iteration. Later in the first day, I became slightly worried that evaluators might no longer be needed for choosing appropriate methods… well, maybe not that worried yet, but the resource being developed by Dr. Barbara Befani and Michael O’Donnell was a really interesting development in actually setting out a list of available methods, let alone how it has drawn on expert feedback to advise on which ones may be most suitable for different situations.
Simple, complex and complicated. CECAN Director Nigel Gilbert introduced the challenges of upward and downward causation, and the example of steering a wheelbarrow to illustrate a wobbly evaluation pathway… and I learned that ‘complex’ is derived from the Latin to ‘intertwine’ (…I’ll probably be using that as a random fact in conversations). Prof. Picciotto also ran a very lively session trying to convince us that the frequently used ‘simple’ example of following a recipe (as opposed to sending a rocket to the moon or raising a child – Glouberman & Zimmerman, 2002) is not even simple, especially when celebrity chefs get involved!
Demonstrating policy influence and intangible added value. This was applicable for a lot of the discussions on how to measure strategic influence in an international development context, and it was a great introduction to process tracing and Bayesian confidence updating (using hoop tests, smoking gun tests, doubly-decisive tests and straw-in-the-wind tests) for assessing confidence in causal attributions (session 9). In an EC context, a similar challenge is being tackled to demonstrate the value of European-level approaches for certain issues and to address calls for subsidiarity (session 2). Carol Candler shared experiences of conducting strategic consultations in Singapore where the emphasis on ‘face’/saving face meant that critical assessments needed sensitivity and delicate handling, but that attempts from strategic leaders to delegate their interviews should be resisted.
Engaging audiences. From Dr Beatriz Garcia’s presentation on cultural / economic impacts of the Liverpool European Capital of Culture 2008, I liked how a pedagogic approach made it simpler to map/present investments allocated to the event itself, wider city regeneration, and meeting wider European objectives; and secondly, how using regular, glossy updates on impact during the event(s) helped promote engagement and strategic buy-in. Claire Hutchings made the good point that we should be moving towards evidence-informed policy, not evidence-based policy.
History of evaluation. Finally, Bradford Rohmer gave an enjoyable presentation on the history of evaluation in the EC, from early written evaluation guidance in the late 1990s which was restricted to mid-term and ex-post; to the inclusion of ex-ante from the 2000s; and the emphasis on DG standardisation and evaluation working documents in last year’s Better Regulation Package.
Sessions attended below:
- Evaluating complexity, Professor Nigel Gilbert, Director, Centre for the Evaluation of Complexity Across the Energy-Environment-Food Nexus (CECAN)
- What is EU-added value and how can it be measured? Andrew Hetherington, Coffey
- Evaluating complexity every day: Practical approaches to evaluating complexity in European funding, Laura Hayward, ICF International
- Evaluating the culture of major events: The long-term view, Dr Beatriz Garcia, Head of Research, Institute of Cultural Capital, University of Liverpool
- Unpacking methodological appropriateness for Impact Evaluation: Presentation of an online tool for selecting appropriate methods, Dr Barbara Befani, University of East Anglia; Michael O’Donnell, Bond
- Understanding what works: Do we know how to mix methods? Professor Bob Picciotto, King’s College London
- Making the infinite countable? Responding to the challenges when evaluating innovation policy, Jonathan Cook, SQW
- youngballymun’s performance story report: A rigorous and pragmatic evaluation of a complex community change initiative, Dr Gemma Cox, youngballymun
- From assessing impact to assessing confidence about impact: Harnessing the potential of Process Tracing and Bayesian confidence updating to evaluate policy influence in complex and uncertain settings, Dr Barbara Befani, University of East Anglia; Gavin Stedman-Bryce, Pamoja UK Ltd; Stefano D’Errico, IIED; Francesca Booker, IIED and Centre for International Forestry Research
- Unpacking and optimising mixed methods evaluation: Insights from the Carers’ Employment Pilot Evaluation, Dr Annette Cox, IES
- Halfway house: The confused past and uncertain future of evaluation in the European Commission, Bradford Rohmer, Coffey International
- UK Evaluation Society AGM
- Recognising messiness and embracing real world complexities: Evaluation and the SDGs, Claire Hutchings, Head of Programme Quality, Oxfam
- Voluntary Evaluator Peer Review update, Derek Poate, Chair, UKES VEPR Sub-group
- Learning from our successes: A positive approach to assessing Public Value, Carol Candler, Voluntary and Philanthropy Sector Development Advisor; Helen Highley, Brightpurpose
- Joining the dots…between services, evaluators and funders: The Project Oracle journey, Professor Georgie Parry Crooke, London Metropolitan University
This year’s full programme is available at: www.profbriefings.co.uk/ukes2016/
Thanks to all the organisers and presenters at this year’s event.