The reliability of ethical review committees

I recently submitted the same application for ethical review for a multi-country study to three ethical review panels, two of which were overseas and one in the UK. The three panels together raised 19 points to be addressed before full approval could be given. Of these 19 points, just one was raised by two committees and none was raised by all three. Given CLAHRC WM’s methodological interest in inter-rater reliability and my own interests in the selection and assessment of health care students and workers, I was left pondering a) whether different ethical review committees consistently have different pass/fail thresholds for different ethical components of a proposed research study; and b) whether others have had similar experiences (we would welcome any examples of either convergent or divergent decisions by different ethical review committees).

Let me explain with two examples. One point raised was the need for formal written client consent during observations of Community Health Workers’ day-to-day activities. We had argued that because the field worker would only be observing the actions of the Community Health Worker and not the client, then formal written client consent was not required, but that informal verbal consent would be requested and the field worker would withdraw if the client did not wish them to be present. The two overseas committees both required formal written client consent, but the UK committee was happy with our justification for not doing so. On the other hand, the UK committee did not think we had provided sufficient reassurance of how we would protect the health and safety of the field worker as they conducted the observations, which could involve travelling alone into remote rural communities. The two overseas committees, however, considered our original plans for ensuring field worker health and safety sufficient.

What are the potential implications if different ethical review committees have different “passing standards”? As with pass/fail decisions in selection and assessment, there could be false positives or false negatives if studies are reviewed by “dove-ish” or “hawk-ish” committees respectively. As with selection and assessment, a false positive is probably the most concerning of the two: a study is given ethical clearance when ethical issues that would concern most other committees have not been raised and addressed. Although it is probably very rare that a study never gets ethical approval, a false negative decision would mean that the research team is required to make potentially costly and time-consuming amendments that most other committees would consider excessive. I have no experience on the “other side” of an ethical review committee, but I expect there must be some consideration of balancing the need for the research findings against potential ethical risks to participants and the research team.

Two interesting research questions arise. The first is to examine how ethical review committees make their decisions and set passing standards for research studies. A study of this nature in undergraduate medical education is currently ongoing: Peter Yates at Keele University is qualitatively examining how medical schools set their standards for finals examinations. The second is to explore the extent of the difference in passing standards across ethical review committees, by asking a sample of committees to each review a set of identical applications and to compare their decisions. A similar study in undergraduate medical education investigated differences in passing standards for written finals examinations across UK medical schools.[1] To avoid significant bias due to the Hawthorne effect, the ethical review committees would really need to be unaware that they were the subjects of such research. This, of course, raises a significant ethical dilemma with respect to informed consent and deception. Therefore it is not known whether such a study would be given ethical approval (and if so, by which committees?).

— Celia Taylor, Associate Professor


  1. Taylor CA, Gurnell M, Melville CR, Kluth DC, Johnson N, Wass V. Variation in passing standards for graduation‐level knowledge items at UK medical schools. Med Educ. 2017; 51(6): 612-20.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s