Re-Solv's Research Blog

Evaluating Prevention: what to measure?

2012-02-12T15:06:00.002Z

As we have seen (in my previous two blogs), partly because of the conceptual difficulties in defining substance misuse prevention and its expected outcomes, it is difficult to evaluate. But evaluate we must, because without an assessment of outcomes we can’t know whether what we are doing is having a worthwhile effect.

Much prevention effort – perhaps most of it – is wasted because it doesn’t appear to have an impact on substance misuse. But perhaps this is because of the difficulties of measuring effects – maybe (for example) there is an impact but it isn’t immediate. A lot of drug prevention aims to prevent drug misuse in the future – but most evaluations are short-term. It’s hard to do long-term evaluations, partly because of the expense and the difficulty of keeping in touch with people over many years, but mainly because the more time that’s passed the more difficult it is to attribute any effects to specific past experiences.

For example, if a group of young people receive alcohol education and immediately afterwards their alcohol consumption declines (assuming a reliable measure of their alcohol use before and after the educational intervention), then it would probably be reasonable to attribute the change to the education. If, at a five-year follow-up, they are using less alcohol than a control group which didn’t receive the alcohol education, then this might be due to the education received five years earlier, but so many other things will have happened in the intervening years that it would be hard to make such a claim.

Another difficulty is the selection of the outcome measurement. Alcohol education doesn’t generally aim to prevent all alcohol use. It might target, for example, binge drinking, drunkenness, or drinking within safe limits. Measuring different expected outcomes might give different results.

A recently-published example is of an evaluation of alcohol prevention which used a sophisticated design, a large sample, and complex statistics to assess the effectiveness of the Örebro Programme (ÖPP), an intervention aimed at reducing youth drinking by changing parental behaviour which is widely used in Sweden (Bodin & Strandberg, 2011 Addiction, 106, 2134–2143). Measured outcomes included frequent drunkenness, lifetime drunkenness, weekly drinking, and servings of alcohol in the home. The authors conclude: “The Örebro prevention programme as currently delivered in Swedish schools does not appear to reduce or delay youth drunkenness.”

But how robust were the outcome measures? In a commentary on the Addiction paper it is pointed out that:

“Bodin & Strandberg summarize a set of behaviours under the outcome of reported frequency of drunkenness, which was dichotomized for analytical purposes, using the following definition of drunkenness: ‘How many times/during the last four weeks/have you drank alcohol to the point that you felt drunk?’. There is a robust body of knowledge challenging the value of reported behaviours such as reported drunkenness, or even comparing different ways to approach self-reports. Moreover, self-perception of drunkenness can vary from individual to individual, influenced by respondent or interviewer characteristics; how much would these perceptions affect the findings? We do not know.” (Pechansky, 2011 Addiction 106, 12, pp 2144–45).

Richard Ives

Conceiving Prevention

2012-02-07T09:09:00.001Z

How do we know if prevention ‘works’? In myprevious blog, I referred to the difficulties of evaluating substance misuseprevention efforts. School-based education for substance misuse preventionremains the leading approach for ‘universal’ prevention efforts, since it canreach all young people, and is relatively inexpensive, but it is bedevilled bydifferent ideas about what should be expected from it and how to go about it.

As the (wonderful) Drug and Alcohol Findings points out in the article, “Drug education yet to fulfil its presumedpotential”:

“School-baseddrug education was and for many remains the great hope for preventing unhealthyor illegal substance use. … The promise is clear, the fulfilment less so.”

Findings points to two issues, firstly:

“… the contradiction between the objectives of education andthose of prevention: the former seeks to empower children to think forthemselves and open up new horizons, the latter to channel thoughts, attitudesand actions in ways intended by programme developers and teachers.”

In other words, prevention tends to aim at stopping people doingthings, but true education is about broadening horizons and developingautonomy. A person well-educated about substances might choose to take drugsand this could be counted as an educational success; but it would not usuallybe seen as a prevention success.

Secondly, Findingspoints to “potentialcontradictions within prevention programmesthemselves”:

“Some aim to limit young people's autonomy in theirchoice of friends and substances by extending autonomy indecision-making, to encourage conformity to non-drug use values by discouraging conformity to other young people, todevelop team work and social solidarity without accepting that youngsters mayexpress this by going along with their peers as well as deciding not to.”

In other words, educators are confused!

The limited impact of substance misuse education on the use ofdrug is perhaps partly explained by these difficulties. There is morediscussion on the Findingswebsite.

Richard Ives

2012-01-06T21:11:00.000Z

Prevention may not bebetter than cure

It’s a truism to say that prevention is better than cure,and in the field of substance misuse it’s clear that treatment is expensive andprevention relatively cheap. ‘Primary’ or ‘universal’ prevention – the sortthat reaches the whole population, such as classes in schools – is especiallycheap, as it can be included in the normal school curriculum and taught byclass teachers or within PSHE (personal, social and health education).

But things are only really cheap if they work. And, forexample, my blog entry of 1^st December pointed out that muchresearch on anti-drug advertising showed that it didn’t have much impact. Thereis no point in spending money on prevention if it doesn’t make a positivedifference.

And there is a risk that some prevention efforts might havenegative effects – the paper referred to in the earlier blog quotes the evaluatorsof the USA’s National Youth Anti-Drug Media Campaign:

‘If the meta-message is that drug use is widespread, higherexposure to Campaign ads should cause an immediate effect on the perceptionthat other kids regularly use marijuana. This perception eventually leads to a more generalised pro-marijuana social norm and greater likelihood of actualinitiation.'

In other words, there is a possibility that banging on aboutdrugs may cause young people to think that their peers are more likely to usedrugs – and therefore they may bemore likely to do so. This violates the maxim ‘do no harm’. (quote from Werb etal 2011 ‘The effectiveness of anti-illicit-drug public-service announcements: asystematic review and meta-analysis’ JEpidemiol Community Health 2011; 65: 834-840).

It’s therefore vital that prevention efforts are properlyassessed for effectiveness.

But this is much easier said than done. Assessingthe outcomes of treatment isn’t easy, either, but at least evaluation oftreatment can study an individual and say whether treatment has been effectiveaccording to criteria such as: are they drug-free; if they are still usingdrugs, are they using them less dangerously, are they committing fewer crimes,and so on. 'Primary'or 'universal' prevention applies to populations, not to individuals, so measurementof effects is more difficult. And outcomes are more varied, and not immediate,having to be looked for over a longer term.

In my next blog entry, I’ll discuss some difficulties in theconception of prevention; after that, I’ll discuss the evaluation of preventionand what can be measured.

Richard Ives

2011-12-20T15:58:00.000Z

Reaching Parents: Acase of successful advertising?

My blog of December 1^st discussed a recent reviewof the evidence on drug prevention advertising campaigns, which was downbeatabout the effects; such campaigns had: ‘a limited impact on the intention touse illicit drugs or the patterns of illicit-drug use among target populations’.

But how about campaigns that aimed at parents? We know thatthe parenting has an enormous impact on all aspects of a child’s life, so itwould not be surprising if parents’ views, attitudes, and behaviours arounddrugs, had a big impact. It’s likely that how they discuss drugs, alcohol andother substances with their children, and the boundaries they establish, willaffect their child’s substance misuse. But we know that parents feel ignorantabout drugs, are worried about discussing substance misuse with their children,and don’t know how to broach the subject. So it makes sense for preventioncampaigns to target parents with drug information, with reassurance, and with ideasfor tackling the issues with their children.

In the 1990s, a Department of Health campaign did this on agrand scale in England and Wales. There were two linked campaigns, one on drugsin general and one specifically on volatile substance misuse. At that time,there was a lot of public concern about VSM and many teenagers (especially boys)were dying from sniffing (in 1990, there were 152 deaths – almost three eachweek). Here’s a contemporary report: http://www.independent.co.uk/news/uk/campaign-to-combat-solvent-abuse-1405249.html

The campaign aimed to get parents talking with theirchildren about substances. It was recognised that TV advertising could have onlylimited goals, and the object of the adverts was to raise parents’ awarenessand encourage them to obtain a copy of the free booklets (one for the drugscampaign and one for the VSM campaign).

The advertisements were hard-hitting – dramaticallyhighlighting VS-related deaths – and probably raised parents’ anxieties; butthey were effective in getting copiesof the booklet into parents’ hands. Did parents act on the sensible andnon-alarmist advice in the booklet? A campaign evaluation indicated that somedid. In the years following the campaign, VSM-related deaths among under-18sdeclined – although how far this was due to the campaign is a difficult question toanswer (we’re trying).

Four of the VSM campaign advertisements can be viewed byfollowing these links:

VSA1: http://www.youtube.com/watch?v=cMaoGVfaVTQ

VSA2: http://www.youtube.com/watch?v=-7SuIDVkMcs

VSA3: http://www.youtube.com/watch?v=lkvAg8e-MRI

VSA4: http://www.youtube.com/watch?v=Xwt_GaVTABM

Richard Ives

What’s in a name?

2011-12-12T20:35:00.000Z

Re-Solv hasrecently started using the term ‘volatile substance misuse’ (‘VSM’) inpreference to the more traditional’ ‘volatile substance abuse’ (‘VSA’). WhenRe-Solv first started out (25 years ago!), the term, ‘glue-sniffing’, wascommonly used, and in those days this was a reasonable description of much ofthe ‘sniffing’. Later, Re-Solv started calling itself ‘the society for theprevention of solvent and volatile substance abuse’, but of course this did notactually mention aerosol or butane misuse. None of these terms communicate verywell to the general public, so choosing the right term is a compromise between somekind of scientific accuracy and a degree of public understanding.

‘VSA’became widely used, but there was always the worry about that word, ‘abuse’.‘Child abuse’ means that the child is being abused – i.e. hurt in some way.Sniffing a commercially available product to achieve intoxication is not‘abusing’ the product, however muchthe ‘sniffer’ is abusing him/herself.

Earlierthis year we had the opportunity to comment on a draft Australian document ofclinical guidelines on VSA management (see previous post). To our dismay, theterm ‘VSU’ (‘volatile substance use’) was used throughout this document. Weresponded to this usage as follows:

‘We are unhappy with the term ‘VSU’ because itconfuses ‘use’ with ‘misuse’. With substances such as crack cocaine that do nothave legitimate uses, the term, ‘use’, may arguably be appropriate. But withsubstances that have legitimate – and safe – uses, and indeed are manufacturedfor such purposes, it seems perverse to describe their misuse to achieveintoxication as ‘use’. While we appreciate that this terminology is intended togive a neutral tone to descriptions of the practice, we fear that it provideslegitimacy. One would expect that we at Re-Solv, a campaigning organisation,would take such a stance. But we do not believe that it is in any waystigmatising those who inhale volatile substances to achieve intoxication todescribe their behaviour as product misuse.’

While they didn’t accept our comment, thinking about thisterminological trickiness helped to propel Re-Solv to take the plunge on a namechange: so it is VSM from now on –but will it stick? (groan)

Richard Ives

2011-12-05T13:47:00.001Z

Australian ClinicalGuidelines

Re-Solv and educariwere recently thanked for their helpful comments on an Australian draftdocument on the clinical management of VSM. The final version of the Document wasformally released by the Minister for Mental Health and Ageing, Mark Butler,and Minister for Indigenous Health, Warren Snowdon on Friday 28 October 2011.

Thepublication, Consensus-based clinicalpractice guideline for the management of volatile substance use in Australiais produced by Australia’s National Health & Medical Research Council.These are the first systematically developed Clinical Practice Guidelinesspecifically developed for Australian health care settings, providingrecommendations to manage volatile substance misuse in Australia. They areintended for health professionals including doctors, nurses, Aboriginal healthworkers, Ngangkari (traditional healers), alcohol and other drug workers andallied health professionals.

TheGuidelines are accompanied by a quick reference guide. This summary containsall the Guidelines’ recommendations in an easy-to-read format to assist healthprofessionals in implementation.

Electroniccopies of the Guidelines and summary document can be accessed at: http://www.clinicalguidelines.gov.au/search.php?pageType=2&fldglrID=1965&

While thePublication takes on many of our comments on their Draft, one that was nottaken on board was our critique of their term ‘volatile substance use’. Why dowe object to this term? See next blog entry.

2011-12-01T17:22:00.001Z

Does advertisingwork?

A recent research paper reviews the effectiveness of campaignsto reduce or prevent drug misuse. The review uses the method of ‘meta-analysis’.This means identifying all the relevant papers in a certain time period, from awide range of journals, and making an assessment of their findings and – crucially– the quality of their findings. The authors conclude:

‘Existing evidence suggests that the dissemination ofanti-illicit-drug PSAs [Public Service Announcements - i.e., advertising‘against’ illicit drug use] may have a limited impact on the intention to useillicit drugs or the patterns of illicit-drug use among target populations.’

(Werb et al 2011 ‘The effectiveness of anti-illicit-drugpublic-service announcements: a systematic review and meta-analysis’ JEpidemiol Community Health 2011; 65: 834-840)

Yet many of us remain convinced that properly designed and executed campaigns can make a difference. Campaignsalso need to be big enough and last long enough to make an impact.

Many different organisations produce prevention material. Inthe case of volatile substance misuse, this sometimes includes the industry.

Whatdo you think of this attempt to discourage young men from sniffing ‘Axe’ deodorant(a USA product)? http://www.youtube.com/watch?v=ubONs4D1abw

Richard Ives

difficult times

2011-10-04T14:55:00.005+01:00

Our research work is progressing - albeit, more slowly than was anticipated in the high hopes of our project planning. This is generally the way with research - one gains a little knowledge but it can be a crawling slog to extract the nuggets of fact from the mountains of 'stuff'.

We are not being helped by the macro-economic conditions: the cuts in services for young people, especially, are affecting our opportunities for reaching professionals who are working with young 'sniffers'. And a common view is that volatile substance misuse has fallen still further down the professionals' agendas.

But there are many bright spots - for example, a recent publication from the Welsh Assembly Government puts VSA at the heart of their Substance Misuse Treatment Framework: the 'Volatile Substance Misuse Module' can be downloaded as a pdf: http://wales.gov.uk/docs/dsjlg/publications/commsafety/110518vsaen.pdf

The Document gives lots of guidance on tackling VSA. It points out that:

VSA-related deaths are proportionately higher than in England
Chronic VSA is linked with a range of other problems and difficulties
Volatile substances are, for many children and young people, the first misused substance.

It advocates that:

Early and sustained prevention activities, including education in school, should take place
It is essential that parents and carers are aware of the risks of the misuse of everyday products
Retailers should be aware of VSA and meet their legal responsibilities.

2010-09-22T13:58:00.001+01:00

I'm excited to be part of this research project which will help Re-Solv develop its support for those working to prevent VSA and assist those who help and treat those with VSA-related problems.

Richard Ives (educari lead researcher)

2010-08-10T10:48:00.002+01:00

Re-Solv's research project is now underway. Our partner organisations educari and St George's University of London have begun work into the current picture of VSA in the UK today. We will bring you updates about the research as and when they become available to us.

Thank you for helping us to keep VSA on the public agenda.

2010-05-14T14:29:00.001+01:00

Re-Solv has started work with our partners educari and St George's University Hospital London to explore current patterns of VS misuse, and to look into what professionals think is currently the most effective prevention.