29 November 2005

Evaluating Quality in Qualitative Research

Carrying on from Ben’s ‘Levels/Typology of Evidence’ post, some recent interesting work on appraising qualitative research and evaluation has been afoot in the UK Cabinet Office (part of the UK. Govt.’s push to encourage evidence informed policy).

Given the importance of qualitative in addition to quantitative evidence for HIA, this UK document is a good introduction into the can of worms that is, ‘What is the quality in qualitative methods?’ Or in other words, given the vast amount of both well and poorly conceived and reported qualitative work out there, how do we choose what is the best available qualitative evidence to assist our HIA?

The original UK document can be downloaded from http://www.strategy.gov.uk/downloads/su/qual/index.htm

At 170 pages the report is (ironic given its subject matter!) very long, but here are its key elements, followed by some caveats and qualifications.

The suggested framework for considering quality in qualitative research is based around: (continue reading)

Four guiding principles - that the research should be:
– contributory in advancing wider knowledge or understanding;
– defensible in design by providing a research strategy which can address the evaluation questions posed;
– rigorous in conduct through the systematic and transparent collection, analysis and interpretation of qualitative data;
– credible in claim through offering well-founded and plausible arguments about the significance of the data generated.

And eighteen appraisal questions:

1. How credible are the findings?
2. How has knowledge or understanding been extended by the research?
3. How well does the evaluation address its original aims and purpose?
4. How well is the scope for drawing wider inference explained?
5. How clear is the basis of evaluative appraisal?
6. How defensible is the research design?
7. How well defended are the sample design/target selection of cases/ documents?
8. How well is the eventual sample composition and coverage described?
9. How well was the data collection carried out?
10. How well has the approach to, and formulation of, analysis been conveyed?
11. How well are the contexts of data sources retained and portrayed?
12. How well has diversity of perspective and content been explored?
13. How well has detail, depth and complexity (i.e. richness) of the data been conveyed?
14. How clear are the links between data, interpretation and conclusions – i.e. how well can the route to any conclusions be seen?
15. How clear and coherent is the reporting?
16. How clear are the assumptions/ theoretical perspectives/values that have shaped the form and output of the evaluation?
17. What evidence is there of attention to ethical issues?
18. How adequately has the research process been documented?

There are also some caveats and qualificationsthat need to be considered:
“The framework is designed to aid the informed judgement of quality, but not to be prescriptive or to encourage the mechanistic following of rules.”
What this means is that qualitative research is a value based exercise, and thus assessment of its quality is necessarily value based also. Use the framework to help you make a judgement about the research, but be flexible and don’t let the framework take over from your personal viewpoint!!
“Most of the items included in the framework are heavily recurrent in the wider literature.”
True, and for those interested it will be invaluable to check out the ‘The bible’ of qualitative methods: Denzin, N. K. and Lincoln, Y. S. (Eds.) (2000). Handbook of qualitative research. 2nd edition. Thousand Oaks, CA: Sage. In particular look at some of the earlier Chapters which go into vital values and perspectives stuff. It will also be useful to search out journals such as ‘Qualitative Health Research’, of ‘Qualitative Inquiry’, and ‘Qualitative Research’. Look directly at how the equivalent of ‘Methods’ and ‘Results’ sections are reported and try to put the criteria to the test!
“Conceptions of quality are influenced by the various philosophical assumptions which underpin different approaches to qualitative research. These epistemological and ontological positions are diverse and span issues such as the nature of reality, the relationship between the researcher and the researched, the relationship between facts and values, the nature of knowledge, and appropriate methods of research.”
Long winded, but true – see the two points above on how to dissect this statement.

Happy hunting!!

By Patrick Harris, CHETRE

22 November 2005

If you can read this you're too...?

Originally uploaded by kacyhsu.
It's come to my attention that some people are having trouble accessing this blog via their work web connection. After some investigation I've found that some (over?)zealous system administrators ban access to any URL with "blog" in it.

Do you think blogs such as this can pose a potential threat to organisations? Apparently the practice is reasonably widespread, as highlighted in this Wired article: (more)
.... companies worry that employees might leak sensitive material -- perhaps inadvertently -- while posting comments to blog message boards. In a survey of over 300 large businesses conducted in conjunction with Forrester, Proofpoint found 57.2 percent of respondents were concerned with employees exposing sensitive material in blogs. That's higher than the portion concerned with the risks of P2P networks.

The upshot of this is that I'm investigating ways of moving this blog to a different URL. I'll keep you all posted on developments.

21 November 2005

UK HIA Conference: Details Online!

The UK & Ireland HIA conference that I mentioned previously has posted further details on the web. You can access the information at:


Are any of you planning on attending?

14 November 2005

Physical Activity and Public Transport

I know that some of the developmental sites are looking at the impact of urban development on physical activity, with reference to public transport. I came across an article that may be of interest. The citation is:

Besser L, Dannenberg A. Walking to Public Transit Steps to Help Meet Physical Activity Recommendations. American Journal of Preventive Medicine 2005; 29(4):273-280

Abstract (more)
Nearly half of Americans do not meet the Surgeon General's recommendation of >= 30 minutes of physical activity daily. Some transit users may achieve 30 minutes of physical activity daily solely by walking to and from transit. This study estimates the total daily time spent walking to and from transit and the predictors of achieving 30 minutes of physical activity daily by doing so.

Transit-associated walking times for 3312 transit users were examined among the 105,942 adult respondents to the 2001 National Household Travel Survey, a telephone-based survey sponsored by the U.S. Department of Transportation to assess American travel behavior.

Americans who use transit spend a median of 19 minutes daily walking to and from transit; 29% achieve >= 30 minutes of physical activity a day solely by walking to and from transit. In multivariate analysis, rail users, minorities, people in households earning < $15,000 a year, and people in high-density urban areas were more likely to spend; >= 30 minutes walking to and from transit daily.

Walking to and from public transportation can help physically inactive populations, especially low-income and minority groups, attain the recommended level of daily physical activity. Increased access to public transit may help promote and maintain active lifestyles. Results from this study may contribute to health impact assessment studies (HIA) that evaluate the impact of proposed public transit systems on physical activity levels, and thereby may influence choices made by transportation planners.