Understanding newspaper headlines about food and medicine

How do we know whether or not to believe a claim about the health effects of a food or medicine, in the papers or on TV?

We can probably start by trying to gain an insight into the authors credentials. Is it evident that they have studied nutrition in depth? Sometimes this is hard to tell unless you have yourself studied nutrition. When it comes to qualifications, these can be claimed falsely or indeed have been earnt rather too easily. The most read nutrition articles are by journalists with very little background in nutritional science. Headlines appear regularly in newspapers and on the TV channels claiming that this or that food or activity impacts on this or that disease. To give three examples from today's BBC health website, we have teeth flossing linked with improved fertility, Champic (varenicline) a quit smoking drug linked to heart problems and mobile phone usage being associated with brain cancer. Normally these headlines have come from scientific studies that have recently been published. The problem is, are the journalists giving an accurate picture of what the study has found? Often they are not and this is compounded by the fact that a large number of scientific studies reach misleading conclusions.

So what we can understand from the scientific studies that are carried out? It is vital for people involved in the field of nutrtion, from journalists to dieticians and nutritional therapists, to understand how to interpret the evidence. To understand the science, a nutritional expert needs to know the different types of scientific study that can be carried out and the limitations that these studies have. An understanding of the types of statistical measurements that show how significant any findings are, is also essential and I will look at this last aspect in a future article.

What types of scientific study are there?

Studies come in two main forms. Observational (epidemiological) and experimental. Both these types have their problems.

Epidemiological studies: These studies appear very often and are reported almost as if they are fact. In reality they are quite limited by the assumptions and skill of those who carry out the studies. In a nutritional context they try and produce associations between diet and various diseases. Examples could be saturated fat intake and heart disease, salt intake and high blood pressure, sweeteners and cancer or even vaccines and autism.

Epidemiological studies try to look at existing groups of people and see if their diets or lifestyle can be linked to the prevalence of disease in that group. Often it is just one dietary factor and one disease that are investigated. These groups are not controlled like laboratory rats however. They are free living human beings, and so what they eat and how they live is complex. As a result trying to associate one food with one disease is riven with difficulty, as many other confounding factors come into play.

Example epidemiological study associating salt with high blood pressure.

A link between salt and high blood pressure could have several confounding factors:

• The measurement of salt intake is difficult to do accurately. Different foods contain different amounts, but may vary depending on their source as well as what type they are. For instance bakery bread can contain mountains of salt, whereas a home made loaf may contain none.
  
• People who eat salty food could also be more likely to eat other nutrients that could raise blood pressure. Equally they could also have lifestyle differences. For instance convenience food is mostly salty and people who eat it are less health conscious, as such they are less likely to exercise and eat plenty of fruit and vegetables. These factors would also raise blood pressure.
  
• Different populations may have different susceptabilities to changes in blood pressure resulting from salt. So if a study linking salt with high blood pressure was carried out on people in one country it may not apply in another. For example black people are on average more prone to high blood pressure. This may be generally true, but again these bland statements often hide important details. For instance there is more genetic difference between a black member of San tribe originating from Namibia and a typical Nigerian, than between a typical Nigerian and a typical Scotsman. Surprising maybe, but true. It should never be forgotten that people are individuals in a very real sense, and extreme care should be taken when treating people as typical members of a group. 
  

• Cohort studies - these studies are essentially epidemiological in nature. You can use criteria called the Bradford Hill criteria to try to tease out stronger evidence of cause and effect in these epidemiological studies.
  
• Instrumental variable analysis - is one way in which epidemiological studies can be improved. It attempts to eliminate the "healthy user" effect. This type of analysis has been most often used in economic models, but is one important way of trying to account for obeserved and unobserved confounding factors in health related epidemiological studies.
• Case Control studies

Experimental studies: Experimental studies normally control the environment of the study subjects. They generally fall into one of the 4 categories below:

Animal experiments - These experiments are carried out mostly on invertebrates such as flies. When it come to mammals mice are used the most with 20 million rats and mice being used each year in the US¹. Animal experiments, apart from being ethically questionable are limited in a number of respects due to the animals being used being fundamentally different to humans.

• The animals involved are not human and therefore will often respond to nutrient and drugs in different ways from humans. For instance chocolate may seem dangerous if you feed it to dogs, not so for humans. If you deprive a human of any vitamin C for long enough they will die, other mammals as they make their own and so would be fine.
• Experimental animals are normally different sizes, move differently and have different metabolic rates when compared with humans. As such energy and nutrition studies, which are often carried out on rats, cannot easily predict what would happen to humans. To move 1 kilo over 1 metre cost more energy for a rat than a human and yet our bipedal motion is less efficient than their four legged movement. Added to this rats burn up fat to keep warm, which we do hardly at all, meaning that when the temperature changes the amount by which  their energy usage changes is likely to differ from ours.

Short-term human experiments

Clinical trials

Intervention studies

A note about bias in the design, publication and reporting of trial results.

Scientific studies may seem authoratitive and trustworthy, but there are at least four reasons why their findings could be suspect.

1. FUNDING. Some studies are more reliable than others due to their funding. For instance drug company funded studies are 20 times more likely to report positive findings than independent studies². On top of this, the drug company studies are 30 times more likely to reach report positive findings in the conclusion. In other words they like to conclude with the positive points.
2. STUDY DESIGN. The reason why drug companies find so many positive findings is partly due to the way they design their studies. If you pick your subjects carefully and control their environment in certain ways, you can do a lot to skew the results in favour of positive findings for a drug. There are countless examples, but let me give you one from the world of athletics. Say I was testing different types of running shoe and their affect on performance using a treadmill. Threre are a lot of factors that could affect the performance that have to be accounted for. For instance heat and humidity in the room, thirst level of the subject, psychological factors such as if anyone is watching, fatigue due to repeated trials, athlete motivation if he /she believes one product is better than another. Although in theory all these factors should be controlled for, most studies can't control them all. This may be due to simple lack of time or money, but it is easy for bias to creep in when money or reputations are involved. All the experimenter has to do is forget to control for some factors and the results can become innaccurate.
3. PUBLICATION BIAS. As well as study design there is the problem of publication bias where studies with negative conclusions don't hit the press. Recently there has been progress in getting drug companies to register their studies for publication before carrying them out, so that they cannot bury bad results so easily. Still with the power and flexibility of study design still firmly in the control of drug companies it is worth being sceptical about most drug company funded studies that come to positive conclusions about the effects of drugs.
4. RESULTS VS CONCLUSIONS. Most commentators, myself included, rarely have time to check the detail of scientific papers. We often look through summaries and conclusions without looking at the trial setup, numerical results and statistical analysis. In practice the best of us probably only look at details when we are particularly bothered about the veracity of the trial because it is favourite issue for us.

Of course the above is not a reason to ignore the results of scientific papers, as they at least grapple with the truth of the matters in hand. This contrasts with most of our tabloid, broadsheet and online journalists who just pontificate based on prejudice rather than spend time looking at what evidence is availlable. Still it is important to be aware that the cloak of respectibility given by quoting a scientific study is not always what it seems.

References:

1. http://en.wikipedia.org/wiki/Animal_testing#Species

2. http://www.drbriffa.com/2007/12/03/scientists-appear-reluctant-to-admit-...