Homeworks academic service


A review of professional literature on assessment and evaluation

Received Feb 1; Accepted Sep 1. This article has been cited by other articles in PMC. Abstract Background Public Health evaluation is essential to understanding what does and does not work, and robust demonstration of effectiveness may be crucial to securing future funding. Despite this, programs are often implemented with poor, incomplete or no evaluation. Public health practitioners are frequently required to provide evidence for the effectiveness of their services; thus, there is a growing need for evaluation guidance on how to evaluate a review of professional literature on assessment and evaluation health programs.

The aim of this study is to identify accessible high-quality, evaluation guidance, available to researchers and practitioners and to catalogue, summarise and categorise the content of a subset of accessible, quality guides to evaluation. Methods We systematically reviewed grey and academic literature for documents providing support for evaluation of complex health interventions. Searches were conducted January to Marchand included academic databases, internet search engines, and consultations with academic and practicing public health experts.

Data were extracted by two authors and sent to the authors of a review of professional literature on assessment and evaluation guidance documents for comments.

Results Our initial search identified unique documents that were screened to identify those that were 1 developed by or for a national or international organization 2 freely available to all 3 published during or after 4 specific to public health. This yielded 98 documents from 43 organisations. Of these, 48 were reviewed in detail. This generated a detailed catalogue of quality evaluation guidance. The content included in documents covers 37 facets of evaluation.

Conclusions A wide range of guidance on evaluation of public health initiatives is available. Time and knowledge constraints may mean that busy practitioners find it challenging to access the most, up-to-date, relevant and useful guidance. This review presents links to and reviews of 48 quality guides to evaluation as well as categorising their content.

This facilitates quick and each access to multiple selected sources of specific guidance. Electronic supplementary material The online version of this article Evaluation guidance, Public health, Review Background Evaluation is foundational to identification, implementation and dissemination of effective and cost-effective interventions.

There have been many calls to ensure and improve evaluation of interventions and initiatives designed to improve public health. Evaluation is essential to understanding what does and does not work and robust demonstration of effectiveness may be crucial to securing future funding.

Evaluation of public health interventions may be complex and expensive, in part, because such interventions are themselves complex [ 4 ]. They also operate in complex demographic and socio-economic contexts [ 4 ].

Nonetheless, when financial resources are scarce, it is crucial that evidence of effectiveness directs selection of interventions. Repeatedly funding ineffective programs not only squanders valuable resources but has opportunity costs resulting from failure to implement potentially beneficial interventions.

Without good evaluation, it is impossible to distinguish between programs that are having a substantial health impact, those that need to be adapted for different populations, those that need to be withdrawn, and those that may be harmful [ 5 ].

Evaluability assessment facilitates decisions by practitioners, commissioners and researchers about what programs most need to be evaluated. These focus on the stage of development or intervention implementation, whether or not the results of the evaluation are likely to lead to changes in policy or practice, how widespread or important effects of an intervention are likely to be i. Such guidance enables evaluation priorities to be identified when funds for evaluations are limited.

When evaluation is undertaken it is vital that it is conducted in a manner that will produce robust answers to the questions addressed. This requires expertise and, not all practitioners may have had adequate training to enable them to undertake evaluations without support. Consequently, public health practitioners and commissioners may feel under-skilled to conduct evaluations [ 7 ].

This highlights the need for high-quality, useable, feely available and practical evaluation guidance on how to evaluate public health programs. Appeals for better evaluation practice led to a proliferation of guidance and advice to support evaluation of public health programs [ 46 ]. The framework includes a series of steps; from engaging stakeholders through to dissemination of findings. Many of these activities are part of routine practice but others are not.

This framework includes a series of standards that are intended to ensure that evaluations are well designed, rigorous, and suitable for purpose.

  1. These documents had a range of aims including. The essential primary health score is represented by the sum of the final average score assigned to each of the four cardinal primary care domains and their subdomains and the average extent of affiliation score.
  2. The Journal of Family Practice 2001; 50 2. Authors were asked if they i considered the summary to be an accurate and good reflection of their document ii if anything was missing and iii if they thought this would be useful to practitioners.
  3. Reference lists of identified papers, and key journals were also scrutinised. Links to the resource and associated resources were also included.

This provides an overview of the phases a review of professional literature on assessment and evaluation processes involved in the development, implementation and evaluation of complex interventions and, in contrast, to an earlier MRC framework [ 4 ], this revised guide provides advice on how to evaluate highly complex programs using a variety of methods — not just controlled clinical trials.

These are just two of many guides to public health evaluation see too, for example, [ 18 — 10 ]. Such guidance has the potential to facilitate the quality of evaluations and increase the number of programs with strong evidence for their effectiveness which, in turn would allow withdrawal of programs that are ineffective or lack evidence of effectiveness.

However, to be useful such guidance has to be accessible to practitioners and commissioners; they need to know which guides to evaluation to use for what purpose. Unfortunately, the many different guides available may be a barrier to identification of relevant guidance. Novice evaluators who turn to the internet for guidance are faced with many choices and have no map describing the content of available evaluation and so may find it difficult to know what guidance to follow.

Some guides provide generic advice on evaluation generally, others on particular types of evaluation e. Some documents are written for academics, or policy makers, or funders, or experienced evaluators. Other guides are topic specific; such as the UK Public Health England frameworks for obesity prevention.

This can be overwhelming for novice evaluators who have no easy way to select appropriate high-quality guidance for particular evaluation projects [ 7 ]. This lack of guidance on how to access and use guidance on public health guidance is clear from conversations with practitioners.

The present study We aimed to review this literature in order to assess the extent of available guides to evaluation relevant to public health interventions and to identify the content of such guidance. We also planned to provide a catalogue of high-quality, readily-accessible guides that would help practitioners navigate this literature. It was not our intention to synthesis this information, merely describe what is in the documents, and to provide a signpost to inform practitioners where information can be found.

We had two specific aims. Specific aims To identify accessible high-quality, evaluation guidance, available to public health researchers and practitioners To summarise and categorise the content of a subset of accessible, quality guides to evaluation.

Methods Search strategy Evaluation guidance documents were identified between January and March using five strategies: Reference lists of identified papers, and key journals were also scrutinised. It was anticipated that a substantial proportion of guidance documents would not take the form of academic papers and so would not be identified using a traditional literature search. To identify as many non-academic documents as possible, we searched the four main a review of professional literature on assessment and evaluation search engines Google, BING, Yahoo, WebCrawler using a modification of the search strategy Additional file 1.

The first 30 pages that were retrieved from each database using each term were screened. The search was conducted by two authors author one and three with guidance and support from author 2.

We included websites, books, journal articles, policy recommendations, educational resources, tools and frameworks that provide support to public health practitioners undertaking evaluations of public health interventions. We excluded all documents and articles in which the aim was to evaluate a specific intervention as opposed to offering advice on how to undertake evaluations.

Papers reporting on the development of questionnaires or assessment scales for specific interventions were also excluded. Inclusion criteria We included all documents, websites, books, journal articles, policy recommendations, educational resources, tools and frameworks that provide support to public health practitioners reviewing or undertaking evaluations of public health interventions.

Development of questionnaires or assessment scales for specific interventions were also excluded. Selecting a subset of evaluation guidance documents Our search identified guidance documents that can be used to support the evaluation of public health programs. This list included books, reports, webpages, and academic articles.

These documents had a range of aims including: From this list of documents, we selected a subset to review and catalogue in detail. Through discussion the authors agreed four selection criteria. First, that a review of professional literature on assessment and evaluation were free and readily available to public health practitioners. Second, given the changing nature of public health practice, we focused on documents written in or after the year Third, to provide a quality indicator we selected documents sourced or created by national or international organisations.

Fourth, and finally, we assessed the relevance of each document to public health. This resulted in a reduced list of 98 documents produced by 25 organisations.

We then reviewed the documents produced be each of these organisations and selected the most comprehensive or recent evaluation guidance, resulting in 25 guides.

So, for these four organisations, more than one guide to evaluation was retained for detailed examination. Summarising the content of selected guides to evaluation Each of the 48 guides were read by two authors. A content template was developed through discussion and short one-page summaries of each of the 48 guides were produced independently by each reviewer. Each summary provided information on the target audience of the guide, its main aim, a short overview of the guide, and strengths and limitations.

Links to the resource and associated resources were also included. The two reviews of each guide were then combined, retaining content from each review and resolving any discrepancies through discussion. The final integrated review was then sent to the original author of the document for verification. Authors were asked if they i considered the summary to be an accurate and good reflection of their document ii if anything was missing and iii if they thought this would be useful to practitioners.

Content categorisation of evaluation guidance documents Once summaries had been completed, two authors independently coded the content of the 48 guides. Initially, a selection of 5 documents were read by each reviewer and their content listed. Content lists were developed based on what was in documents, rather than according to a pre-existing checklist. Merging of these two content lists through discussion resulted in a set of 44 content categories.

The two authors then jointly coded a further 5 documents, and discussed and refined the list accordingly. The final list contained 37 content categories generated by the initial two-stage coding of 10 guides; which was then used to categorise the content of the remaining 38 guides.

Results Overview of 98 guides to evaluation While we identified guides to evaluation only 98 were relevant to public health, free and readily available and produced since by a national or international organisation. These 98 varied in terms of purpose, topic or condition, and audience. The large majority were general overviews of evaluation focusing on principles of evaluation and how to assess evidence to support evaluation.

National Institute for Learning Outcomes Assessment |

A variety of evaluations were considered including, trials, naturally-occurring experiments, process evaluations and economic evaluations and various advice and instructions were provided on how to plan prepare for and conduct such evaluations.

The guides were mainly generic but some focused specifically on evaluating international development, obesity, asthma, sexual health, mental health and physical activity. One focused on children and families, one on healthy eating and one on drugs and one on violence on women and girls. However, even condition-specific guides provided instruction and support that relevant to evaluations of other types of interventions. For example, some guides, focused on evaluating interventions relevant to particular health problem asthma, smoking and obesity to illustrate more general lessons.

Response from authors Of the 48 documents we reviewed, copies of our review were sent to the lead authors of each guide.