Evaluating Quality in Qualitative Research

Carrying on from Ben’s ‘Levels/Typology of Evidence’ post, some recent interesting work on appraising qualitative research and evaluation has been afoot in the UK Cabinet Office (part of the UK. Govt.’s push to encourage evidence informed policy).

Given the importance of qualitative in addition to quantitative evidence for HIA, this UK document is a good introduction into the can of worms that is, ‘What is the quality in qualitative methods?’ Or in other words, given the vast amount of both well and poorly conceived and reported qualitative work out there, how do we choose what is the best available qualitative evidence to assist our HIA?

The original UK document can be downloaded from http://www.strategy.gov.uk/downloads/su/qual/index.htm

At 170 pages the report is (ironic given its subject matter!) very long, but here are its key elements, followed by some caveats and qualifications.

The suggested framework for considering quality in qualitative research is based around: (continue reading)

Four guiding principles - that the research should be:
– contributory in advancing wider knowledge or understanding;
– defensible in design by providing a research strategy which can address the evaluation questions posed;
– rigorous in conduct through the systematic and transparent collection, analysis and interpretation of qualitative data;
– credible in claim through offering well-founded and plausible arguments about the significance of the data generated.

And eighteen appraisal questions:

1. How credible are the findings?
2. How has knowledge or understanding been extended by the research?
3. How well does the evaluation address its original aims and purpose?
4. How well is the scope for drawing wider inference explained?
5. How clear is the basis of evaluative appraisal?
6. How defensible is the research design?
7. How well defended are the sample design/target selection of cases/ documents?
8. How well is the eventual sample composition and coverage described?
9. How well was the data collection carried out?
10. How well has the approach to, and formulation of, analysis been conveyed?
11. How well are the contexts of data sources retained and portrayed?
12. How well has diversity of perspective and content been explored?
13. How well has detail, depth and complexity (i.e. richness) of the data been conveyed?
14. How clear are the links between data, interpretation and conclusions – i.e. how well can the route to any conclusions be seen?
15. How clear and coherent is the reporting?
16. How clear are the assumptions/ theoretical perspectives/values that have shaped the form and output of the evaluation?
17. What evidence is there of attention to ethical issues?
18. How adequately has the research process been documented?

There are also some caveats and qualificationsthat need to be considered:

“The framework is designed to aid the informed judgement of quality, but not to be prescriptive or to encourage the mechanistic following of rules.”

What this means is that qualitative research is a value based exercise, and thus assessment of its quality is necessarily value based also. Use the framework to help you make a judgement about the research, but be flexible and don’t let the framework take over from your personal viewpoint!!

“Most of the items included in the framework are heavily recurrent in the wider literature.”

True, and for those interested it will be invaluable to check out the ‘The bible’ of qualitative methods: Denzin, N. K. and Lincoln, Y. S. (Eds.) (2000). Handbook of qualitative research. 2nd edition. Thousand Oaks, CA: Sage. In particular look at some of the earlier Chapters which go into vital values and perspectives stuff. It will also be useful to search out journals such as ‘Qualitative Health Research’, of ‘Qualitative Inquiry’, and ‘Qualitative Research’. Look directly at how the equivalent of ‘Methods’ and ‘Results’ sections are reported and try to put the criteria to the test!

“Conceptions of quality are influenced by the various philosophical assumptions which underpin different approaches to qualitative research. These epistemological and ontological positions are diverse and span issues such as the nature of reality, the relationship between the researcher and the researched, the relationship between facts and values, the nature of knowledge, and appropriate methods of research.”

Long winded, but true – see the two points above on how to dissect this statement.

Happy hunting!!

By Patrick Harris, CHETRE

If you can read this you're too...?

cold
Originally uploaded by kacyhsu.

It's come to my attention that some people are having trouble accessing this blog via their work web connection. After some investigation I've found that some (over?)zealous system administrators ban access to any URL with "blog" in it.

Do you think blogs such as this can pose a potential threat to organisations? Apparently the practice is reasonably widespread, as highlighted in this Wired article: (more)

.... companies worry that employees might leak sensitive material -- perhaps inadvertently -- while posting comments to blog message boards. In a survey of over 300 large businesses conducted in conjunction with Forrester, Proofpoint found 57.2 percent of respondents were concerned with employees exposing sensitive material in blogs. That's higher than the portion concerned with the risks of P2P networks.

The upshot of this is that I'm investigating ways of moving this blog to a different URL. I'll keep you all posted on developments.

UK HIA Conference: Details Online!

The UK & Ireland HIA conference that I mentioned previously has posted further details on the web. You can access the information at:

http://www.wales.nhs.uk/sites3/news.cfm?orgid=369&contentid=4036

Are any of you planning on attending?

Physical Activity and Public Transport

Maggie takes a walk @ Roseville REI
Originally uploaded by tromasbronot.

I know that some of the developmental sites are looking at the impact of urban development on physical activity, with reference to public transport. I came across an article that may be of interest. The citation is:

Besser L, Dannenberg A. Walking to Public Transit Steps to Help Meet Physical Activity Recommendations. American Journal of Preventive Medicine 2005; 29(4):273-280

Abstract (more)
Background
Nearly half of Americans do not meet the Surgeon General's recommendation of >= 30 minutes of physical activity daily. Some transit users may achieve 30 minutes of physical activity daily solely by walking to and from transit. This study estimates the total daily time spent walking to and from transit and the predictors of achieving 30 minutes of physical activity daily by doing so.

Methods
Transit-associated walking times for 3312 transit users were examined among the 105,942 adult respondents to the 2001 National Household Travel Survey, a telephone-based survey sponsored by the U.S. Department of Transportation to assess American travel behavior.

Results
Americans who use transit spend a median of 19 minutes daily walking to and from transit; 29% achieve >= 30 minutes of physical activity a day solely by walking to and from transit. In multivariate analysis, rail users, minorities, people in households earning < $15,000 a year, and people in high-density urban areas were more likely to spend; >= 30 minutes walking to and from transit daily.

Conclusions
Walking to and from public transportation can help physically inactive populations, especially low-income and minority groups, attain the recommended level of daily physical activity. Increased access to public transit may help promote and maintain active lifestyles. Results from this study may contribute to health impact assessment studies (HIA) that evaluate the impact of proposed public transit systems on physical activity levels, and thereby may influence choices made by transportation planners.

Ethics Approval and HIA

The 'Ethics and Disobedience' strand
Originally uploaded by bowbrick.

A number of people have been wondering if HIAs require approval from institutional ethics review committees in order to be undertaken (as you'll recall from the training). I don't think it's appropriate to emphatically rule out the need for ethics approval but there is clearly a strong case to be made that your HIAs are being undertaken for quality assurance/quality improvement purposes, rather than as research. After all, the goal of your HIA is to recommend improvements to the proposal it's being conducted on.

The NHMRC has a number of publications on the ethical conduct of research involving humans that may be of interest. Their National Statement on Ethical Conduct in Research Involving Humans states that: (continue reading)

There are many definitions of research. These include systematic investigation to establish facts, principles or knowledge and a study of some matter with the objective of obtaining or confirming knowledge. A defining feature of research is the validity of its results. The knowledge that is generated by research is valid in the sense that what is discovered about the particular facts investigated can be justifiably claimed to be true for all like facts. However, it remains difficult to find an agreed definition of research.

An alternative approach to finding a definition of research is to list examples of what constitutes research, such as:

- systematic prospective collection of information to test an hypothesis;
- a planned study of existing practices with a view to changing/improving practice in light of the study’s findings and/or to increase understanding; or
- the administration and analysis of data in response to surveys or questionnaires, interviews or opinion polling.

However, such lists risk including activity that would not normally be included, like quality assurance activities [my emphasis] or audits and excluding activity that probably
should be included, such as research conducted as part of a course of education.

Given that HIAs (1) don't seek/promise validity of their findings beyond the specific proposal being considered and (2) are not testing a hypothesis per se, it may not be appropriate to view them as research in the sense described in the NHMRC Statement. I think it depends on your own judgement, work context and circumstances to determine whether an ethics approval is required.

What are your thoughts?

Public Participation Spectrum

I've had a few queries about the public participation spectrum mentioned in the training materials. The spectrum was developed by the International Association for Public Participation (IAP2) and range of public involvement that is possible, from informing to empowering.

Participation Spectrum (click to download PDF)

You can download the spectrum here (PDF 46Kb). This may be relevant for your HIA for two reasons: (more)

• Determining how involved the affected communities should be in conducting the HIA, and
• Determining how information will be collected from the affected communities.

Scoping: A question of values?

One of the most crucial aspects of scoping is reaching agreement about how evidence will be valued. Practitioners report that this is often overlooked and then becomes a problem during the "assessment of impacts" step.

Deciding on how to value evidence will also help to guide the rest of your scoping. If a traditional hierarchy of evidence is going to be used (see diagram below) there's not much point in collecting "lay opinion".

A hierarchy of evidence (click to enlarge)

But if a typology of evidence approach (below) is used community views may be important in aswering questions of appropriateness and acceptability. What is often dismissed as "lay opinion" may be quite valuable because it (a) comes from a different knowledge and values base than other types of evidence, (b) is important in determining outcomes and (c) may offer unique insights.

A typology of evidence (click to enlarge)
NB: both digrams are from Petticrew M, Roberts H. Evidence, Hierarchies and Typologies: horses for courses, Journal of Epidemiology & Community Health, 57:527-9, 2003. Link to Fulltext)

It's also important to realise that agreeing on how evidence will be valued doesn't predetermine what the assessed impacts will be. It's about ensuring clarity of purpose:

• Will you HIA have a broad of tight focus?
• Will it focus on health protection, health promotion or both?
• Will it be used as a political or scientific tool?

Resources that might be of use: (more)

NHMRC. Using Socioeconomic Evidence in Clinical Practice Guidelines, National Health & Medical Research Council: Canberra, 2002.

This handbook provides information to assist developers of clinical practice guidelines (CPGs) in summarising the evidence of the effects of socioeconomic position (SEP) and other markers (or descriptors) of disadvantage, on health outcomes and health care delivery. This process informs the development of CPGs by suggesting ways in which guideline developers can identify and retrieve evidence and incorporate it into guidelines for clinical practice.

Weightman A, Sander L, Turley R, Ellis S, Cullum A. Grading evidence and recommendations for public health interventions: Developing and piloting a framework, Support Unit for Research Evidence (SURE), Information Services, Cardiff University & the Health Development Agency, London, 2005.

“This provisional framework provides a practical and transparent method for deriving grades of recommendation for public health interventions, based on a synthesis of all relevant supporting evidence from research.”

"..The framework was to relate only to the grading of evidence and recommendations for public health interventions. (The types of evidence that are relevant to other (non-intervention) aspects of public health will be included in further developments of the methodology.) Development of the framework has incorporated an analysis of the published literature on deriving grades of evidence and recommendations for public health interventions, and consultation with public health and methodology experts.

UK & Ireland HIA Conference 2006

UK
Originally uploaded by IanL.

The UK & Ireland Health Impact Assessment Conference is being held in Cardiff on the 4th, 5th and 6th of April, 2006. The email address to contact is hia2006@cardiff.ac.uk.

I know it's a long way off, in both time and distance terms, but I thought I should let you know in case anyone was planning a trip to Wales. I was fortunate enough to go to it in 2004 and it's a very worthwhile conference.

Update: The conference website is now up.

RIVM Health Impact Assessment Database

A new HIA website has been set up by the Netherlands Public Health Institute. There's not much material up now but it may become a useful source of completed HIA reports and grey literature in the future.

RIVM Health Impact Assessment Database

Diary Date - Friday, 9th of December 2005!

Diary
Originally uploaded by Auntie P.

As you are hopefully all aware CHETRE and NSW Health are organising a colloquium on HIA to be held in Sydney on Friday the 9th of December 2005.

I hope you're all able to attend, and more importantly submit abstracts for presentations on your HIAs.

Further details are available on HIA Connect. You can download the colloquium flyer here.

Screening: What are the issues?

Work to be done
Originally uploaded by teckbrain.

So most of you are in the stages of drafting or finalising your screening reports. What are the issues you've encountered so far? What worked and what didn't?

In particular, what resources provided a way forward? Please include links if you have them.

HIA Special Issue of Env. Impact Assessment Review

Purple Reports
Originally uploaded by Nick Dimmock.

A new issue of the journal Environmental Impact Assessment Review has come out dealing entirely with HIA. The articles are based on a number of presentations given at the UK & Ireland HIA conference last year(link to conference presentations, covered in Issue 12 of the HIA E-News).

I have online access to the journal so if you're interested in any of the articles please email me. The issue's contents are: (more)

HIA — Growth and prospects
John Kemm

HIA, the next step: Defining models and roles
Kim Putters

Health impact assessment in multinationals: A case study of the Royal Dutch/Shell Group
Martin Birley

HIA and pollution prevention control: What they can learn from each other
Balsam Ahmad, Tanja Pless-Mulloli and Catherine Vizard

Can health feasibly be considered as part of the planning process in Scotland?

Martin Higgins, Margaret Douglas and Jill Muirie

Canadian insights: The challenges of an integrated environmental assessment framework
Karen McCaig

Making effective links to decision-making: Key challenges for health impact assessment
Eva Elliott and Sarah Francis

Evaluating the impact of HIA on urban reconstruction decision-making. Who manages whose risks?
Marleen P.M. Bekker, Kim Putters and Tom E.D. van der Grinten

Equity-focused health impact assessment: A tool to assist policy makers in addressing health inequalities
Sarah Simpson, Mary Mahoney, Elizabeth Harris, Rosemary Aldrich and Jenny Stewart-Williams

A prospective health impact review of the redevelopment of Central Manchester Hospitals
Neil Bendel and Vicci Owen-Smith

Developing a framework to assess costs and benefits of Health Impact Assessment
Phillip Atkinson and Anthea Cooke

The future challenges for HIA
John Kemm

Why have a blog on HIA?

This blog is intended as a mechanism to encourage information sharing between HIA practitioners and those with an interest in HIA. The project was originally established as part of the NSW HIA Project and is maintained by the Healthy Public Policy Program at the Centre for Health Equity Training, Research and Evaluation (CHETRE), which is part of the UNSW Research Centre for Primary Health Care and Equity.

Blogs allow users like you to interact to a greater degree than traditional websites. You can post comments and other users can respond to what you say. You can inlude links to material you've found or just give your opinion on what has been written.

It's my hope that this blog will be more about what you want to say than what I do. So go on - comment!

Subscribing Using Feeds

What is a Blog Feed?

A blog feed is a data feed produced by a weblog or sites that use feeds (e.g. most newspaper sites). It is designated by a web address, or URL, and is usually in an Extensible Markup Language-variant format like Rich Site Summary (RSS) or Atom.

Why Bother Subscribing?

Subscribing to a feed allows you to receive all new posts without having to visit the site. In essence, it allows you to find out when the blogs you subscribe to are updated and automatically downloads their posts. If you regularly look at a few different sites this is the most time efficient way of staying up to date with their content.

How do I Subscribe?

The easiest way to subscribe to a feed is using a web-based news aggregator:

• Google Reader
• Bloglines
• Newsgator
• My Yahoo

These allow you to easily access a number of different feeds from a single page in your web browser. You simply enter the URL for one of the feeds below into the news reader and you're off and running!

HIA Connect Blog Feeds
http://feeds.feedburner.com/HiaConnectBlog