Development is underway again on Substance Views after the Christmas break, and we’re building up towards the first pilot projects in March. In the meantime, we asked Matthew Pike, Chair of the Substance Views Partnership, and Chair of the Commission on Personalisation, to share his reflections on how evaluation data is key to driving change in public services.

You can register to take part in a Substance Views Pilot here.

It is the first week back after a prolonged Christmas hibernation. As I switch on my computer, second cup of strong coffee in hand, I reflect on the privilege of living at a time when everything is on the move and everything, for good and ill, is possible.

It is rather like looking out on a building site where a series of grand old frontages have been brought crashing down. Top-down state control has failed. Free markets have failed. Great brands have been humbled. Financial services are in disarray. Our public finances are dire. Here we are arriving in 2010 and it is as if the dust has had time to settle: a huge tract of land suddenly looms out at us, ripe for re-development. This is the stuff that gets an entrepreneur’s adrenalin flowing more surely than any number of shots of espresso.

My own focus is public services, where we are seeing nothing less than a wholesale revolution. Political parties of all main persuasions now agree that the future lies in personal control, self-help and mutual aid. Social care, long term medical conditions, unemployed people on incapacity benefit, adult learning… the number of public service areas that lend themselves to forms of personal budgets and user-control is mounting, just as the number of major government pilots has proliferated to 13, impacting on a spend in excess of £28bn and rising.

I am chairing a Commission on Personalisation that is reviewing the practical implications of the move away from block contracts towards forms of personal budgetary control (ranging from cash control to forms of direction or priority-setting by users). Our interim report can be found at http://www.acevo.org.uk/Page.aspx?pid=1167.

One of our key insights is that what is entailed in the move to personalisation is a complete system change. We cannot fiddle around the edges: we need a coherent plan of action that addresses key issues relating to supply, demand, self-help and mutual aid, quality assurance and regulation, information and advice, investment and strategic direction. We need, in short, to build a new system and we haven’t got long. By 2011 many local authorities will have devolved 60-100% of budgets to people eligible for social care funding.

Much – but not all – the answer to the system change challenge ahead lies in smarter use of information technology.

• Use of Visa or Mastercard-based payment systems will cut transaction costs by 20% + and offer increased customer protection.
• Freeing use of public data will generate rich new sources of market intelligence
• Smart brokerage systems like Ebay will allow supply and demand to meet in new ways
• User rating and feedback systems will provide infinitely richer sources to support informed decision-making

At the heart of these challenges, is the requirement for providers to demonstrate value and effectiveness as never before. Here Substance Views has the potential to fill a major gap, by offering not just real-time impact data and forms of analysis but also a rich array of user-generated feedback that can help a million new mini-commissioners, armed with personal budgets, to design their own futures with confidence.

The promise of the reforms that lie ahead is that they can deliver the same or better outcomes with 10-20% lower budgets. Of course this is music to a politician’s ears, but in the end, as ever, it comes down to practicalities. Ultimately we will only succeed if we can provide a service that allows better services and better outcomes to drive out inferior services and inferior outcomes over time: a kind of Gresham’s law in reverse. If is for this reason that the Substance Views programme, as well as associated work on outcomes metrics and rating is of the most critical importance.

Matthew Pike

Chair, Commission on Personalisation
Chair, Substance Views partnership

We’ve spent a long time planning Substance Views – but we’re pleased that development of the new platform is now well under way.

Over the coming months our development partner, Nemisys, will be putting together all the core modules of the new Views platform – taking our learning from SPRS (The Substance Project Reporting System), but looking to create a more dynamic and flexible framework which new inputs and reports can added to with ease.

We’ll be testing each core module of the new system with a small group of partners as the modules are developed – but later in the year we’ll also be inviting a wide range of organisations who have projects that are monitored and evaluated to get involved in a pilot phase of the project. Participants in the pilot will get free access to the developing Substance Views platform during the pilot phase, and will be invited to suggest and discuss the features they most want to see build on top of Views to make it a platform that will work for them.

If you are interested in being part of the pilot, then you can register your interest now.

When we started out on the journey to build the Substance Project Reporting System (SPRS) we were keen to create a really simple to use system with an intuitive interface which would not present itself as a burden on users. In the first instance, working with a single programme for which we determined the parameters of the monitoring requirements, we were largely able to achieve our aim. However, as new projects came on board, they each brought fresh ideas and new requirements with them. As we implemented the various requests, the functionality of the system was greatly enhanced but sometimes at the expense of elegance and simplicity of navigation. In turn, individual programmes sought to restore simplicity through requests for the exclusion of some functions which has resulted in the creation of over 50 separate ‘versions’ of the SPRS to date!

It was in this light that we decided to take a step back and review the way we were developing which also sought to take account of:

•    our awareness of wider changes sweeping through the web based technology sector
•    the demands placed on projects by requirements to use multiple data capture and reporting mechanisms
•    our desire to engage system developers in a more open process of mutually beneficial sharing of data and functionality
•    the need to more clearly and robustly demonstrate programme outcomes in the context of the emergence of outcomes based commissioning regimes

So rather than continue to develop the existing system on an incremental basis we are now working towards the release of a new monitoring and evaluation platform which will include the core functionality that all existing users of the SPRS benefit from around:

•    Contacts – people, groups or organisations
•    Work – done with those contacts
•    Evidence – of the work done with those contacts
•    Reports – on the evidence of the work done with those contacts

As well as ensuring a more usable interface this platform will also facilitate greater flexibility in terms of the ongoing open development of a potentially endless range of third party data collection, outcome monitoring and reporting applications which will be seamlessly integrated with the core platform. This approach will enable user led bespoke configurations of the tools alongside programme level mapping and visualisation of evidence against existing and new national indicators and outcome frameworks.

Ultimately, with the backing of the National Endowment for Science Technology and the Arts (NESTA), the Office of the Third Sector and the Treasury Invest to Save Budget delivered via a partnership with the London Borough of Camden Substance Views will help to realise a new model of outcomes based commissioning, whereby:

•    Commissioners present outcome and data requirements rather than having to specify outputs and procure system solutions.
•    Delivery agencies focus on developing the most effective ways to generate those outcomes rather than ticking boxes and milestones.
•    Data capture mechanisms are embedded in local, routine and ‘real time’ project delivery, management and appraisal processes rather than reporting cycles.
•    Delivery agencies have access to a single flexible data capture and reporting platform which enables reporting in multiple formats.
•    Resources are driven back into projects demonstrating desired outcomes.

Keep following the blog to join our journey, and to input along the way as we work in the open both to explore the intellectual case and context of outcomes based commissioning, and the practical development of the new Substance Views system.

Outcomes Based Commissioning and the Circle of Innovation

So how did a bunch of social science academics with an interest in qualitative research methodologies end up developing monitoring and evaluation software which generates the performance management profiles for a raft of government funded social policy initiatives? Well, funnily enough, the development of the Substance Project Reporting System (SPRS) was initially driven by a need to find ways to get more people to start using the kind of qualitative research methods and evidence that we deployed as professional researchers.

Born of frustration in our inability to scale the intensive qualitative case study research work we were doing with a number of Positive Futures projects as well as the reluctance of the formal university sector to innovate and engage with such evidence, Substance was founded with a mission to transform the nature of social impact evaluation.

As more people became interested in our work we quickly realised that technology had a key role to play in helping us to find sustainable and scalable solutions to the challenges we sought to address and particularly people’s alienation from the process of monitoring and evaluation. In September 2006, a year’s consultation and development culminated in the release of the SPRS which provides a comprehensive online monitoring, evaluation and reporting framework that neatly combines quantitative and qualitative data entry to enable organisations to capture and represent the full range of their work. Now being used by several hundred locally based delivery organisations working in a variety of policy contexts, the SPRS has evolved beyond all recognition and has helped to sustain investment through a series of national Government and third sector policy delivery programmes.

This success is largely attributable to the data generated through users’ engagement with the tools which we would relate to a shifting understanding of the monitoring and evaluation process as having real local purpose rather than being an externally imposed exercise with unclear benefits. This has been achieved through a user centred approach to development which resulted in:

1.    Syncing of the process of monitoring and evaluation with wider project management activity
2.    Enabling the collation, storage and meaningful representation of a wide variety of qualitative evidence
3.    Mapping of multiple forms of data against Government policy outcome frameworks
4.    Giving real time access to data and the ability to analyse it via the user interface

However, despite the brownie points we have earned through the maintenance of a user centred approach towards our ongoing development of the system and our responsiveness to client requests it became clear that this approach can also impact on the underlying principles which informed the initial development. Ultimately this realisation led us to the conclusion that the system must be regarded as the market tested prototype for our new venture, Substance Views!

The Substance Views Online Network is a space for practitioners, funders, commissioners, technologists and other interested parties to come together to explore monitoring, evaluation and outcomes-based commissioning.

Read more »

In late 2008 we held a series of events across the country called ‘The Great Monitoring and Evaluation Debates. Through dialogue with practitioners, funders, commissioners and innovators we explored the tensions and challenges in current models of Monitoring and Evaluation and looked at how, in a world where technologies to record are ubiquitous, new models of collecting and analysing information about how projects are working, and what outcomes they are producing, could be possible. Read more »