Tag Archives: Employment

An Extremely Interesting Three Way Experiment

News Blog readers know that the CLAHRC WM Director is always on the look-out for interesting randomised trials in health care and elsewhere. He has David Satterthwaite to thank for this one – an RCT carried out among applicants for low level jobs in five industries in Ethiopia.[1] The applicants (n=1,000), all of whom qualified for the job on paper, were randomised to three conditions:

  1. Control;
  2. Accepted into the industrial job;
  3. Given training in entrepreneurship and about $1,000 (at purchasing power parity).

Surprisingly, the industrial jobs, while producing more secure incomes, did not yield higher incomes than the control group and incomes were highest in the entrepreneur group. On intention-to-treat analysis the industrial jobs resulted in worse mental health than experienced in the entrepreneurial group, and physical health was also slightly worse. Many left the jobs in firms during the one year follow-up period. In qualitative interviews many said that they accepted industrial jobs only as a form of security while looking for other opportunities.

The authors, aware that rising minimum wages or increasing regulations have costs to society, are cautious in their conclusions. The paper is interesting nevertheless. The CLAHRC WM Director would like to do an RCT of paying a minimum wage vs. a slightly higher wage threshold to determine effects on productivity and wellbeing, positing an effect like this:

065-dcv-fig-1

— Richard Lilford, CLAHRC WM Director

Reference:

  1. Blattman C & Dercon S. Occupational Choice in Early Industrializing Societies: Experimental Evidence on the Income and Health Effects of Industrial and Entrepreneurial Work. SSRN. 2016.
Advertisements

Do we Need ‘Situations’ to Make a Situational Judgement Test?

Rank the following options in order of their likely effectiveness or the extent to which they reflect ideal behaviour in a work situation.

  1. Make a list of the patients under your care on the acute assessment unit, detailing their outstanding issues, leaving this on the doctor’s office notice board when your shift ends and then leave at the end of your shift.
  2. Quickly go around each of the patients on the acute assessment unit, leaving an entry in the notes highlighting the major outstanding issues relating to each patient and then leave at the end of your shift.
  3. Make a list of patients and outstanding investigations to give to your colleague as soon as she arrives.
  4. Ask your registrar if you can leave a list of your patients and their outstanding issues with him to give to your colleague when she arrives and then leave at the end of your shift.
  5. Leave a message for your partner explaining that you will be 30 minutes late.

053 GB - SJT Doctor

How would your ranking change if you knew the following about the situation?

You are just finishing a busy shift on the Acute Assessment Unit (AAU). Your FY1 colleague who is due to replace you for the evening shift leaves a message with the nurse in charge that she will be 15 to 30 minutes late. There is only a 30 minute overlap between your timetables to handover to your colleague. You need to leave on time as you have a social engagement to attend with your partner.

(Example from UKFPO SJT Practice Paper © MSC Assessment 2014, reproduced with permission.)

The use of situational judgement tests (SJTs) for selection into education, training and employment has proliferated in recent years, but there remains an absence of theory to explain why they may be predictive of subsequent performance.[1] The name suggests that the tests are an assessment of a candidate’s ability to make a judgement about the most appropriate action in challenging work-related situations; suggesting that the tests must include descriptions of such challenging work-related situations. But your ranking of the possible actions listed above probably did not change much (if at all) once you knew the exact details of the situation compared to when these had to be deduced from the possible actions listed. A similar finding was recently reported in a fascinating experiment conducted by Krumm and colleagues,[2] with volunteers randomised to complete a teamwork SJT with or without situation descriptions. Those given the situation descriptions scored, on average, just 8.5% higher than those not given the descriptions. Of course, consideration of the need for a situation description is only possible for SJTs in a format where possible actions are presented to candidates (commonly known as multiple choice), but this format is generally used in practice as it facilitates marking and scoring.

Krumm et al.’s findings clearly raise doubts as to the intended construct of the test (i.e. the candidate’s judgement of specific situations); yet SJTs are predictive of workplace performance, with correlations of around 0.30 reported in meta-analyses (see for example McDaniel et al.).[3] So if a SJT doesn’t actually require a “situation” to enable a useful assessment of a candidate’s likely future performance, then what exactly is the assessment of? Lievens and Motowildo [4] suggest that it is of general domain knowledge regarding the utility of expressing certain traits, such as agreeableness, based on the knowledge that such traits help to ensure effective workplace importance. The implication of this theory for practice is that SJTs may not need to be particularly specific and could therefore be shared across professions and geographical boundaries, making them a particularly cost-effective selection tool. The implication for research is that we need more evidence on the antecedents of general domain knowledge, such as family background, both as part of theoretical development and to evaluate the fairness of SJTs for selection.

And what if one does actually desire an assessment of situational judgement as opposed to general domain knowledge, since both have independent predictive validity for job performance? Rockstuhl and colleagues suggest that candidates need to be asked for an explicit, open-ended judgement of the situation (e.g. “what are the thoughts, feelings and ideas of the people in the situation?”) rather than what they think is the most appropriate response to it.[5] The nub here is whether including open-ended assessments to enable measurement of situational judgement is cost-effective given their incremental validity over general domain knowledge and the cost of marking responses (with at least two markers required). For the moment we simply note that a rather large envelope would be required for even a rapid assessment of selection utility!

— Celia Taylor, Senior Lecturer

References:

  1. Campion MC, Ployhart RE, MacKenzie Jr WI. The state of research on situational judgment tests: a content analysis and directions for future research. Hum Perform. 2014; 27(4): 283-310.
  2. Krumm S, Lievens F, Hüffmeier J, et al. How “situational” is judgment in situational judgment tests? J Appl Psychol. 2015; 100(2): 399-416.
  3. McDaniel MA, Hartman NS, Whetzel DL, Grubb III WL. Situational judgment tests, response instructions, and validity: a meta‐analysis. Pers Psychol. 2007; 60(1): 63-91.
  4. Lievens F, & Motowidlo SJ. Situational judgment tests: From measures of situational judgment to measures of general domain knowledge. Ind Organ Psychol. 2016: 9(1): 3-22.
  5. Rockstuhl T, Ang S, Ng KY, Lievens F, Van Dyne L. Putting judging situations into situational judgment tests: Evidence from intercultural multimedia SJTs. J Appl Psychol. 2015; 100(2): 464-80.

Who Would Want to Work in a University?

Many people like to be self-employed – they don’t have to defer to a boss they might not like, they are independent, and they can make their own decisions. But make no mistake, they pay a high price for these freedoms – people working independently earn about 25% less than matched counterparts in employment.[1] Those who leave their companies to start consultancies likewise experience a drop in earnings. And the freedoms they gain are only partial – they are heavily constrained by their clients and the commissions they can secure. Of course, they have to do all their maintenance – IT, pension, legal advice, accountancy, etc. Working as an academic in a research-intensive university provides the best of both worlds. Considerable discretion over which projects to pursue. Opportunities to make strategic and tactical decisions and to display entrepreneurship. The opportunity to secure funding and build a small team. Opportunities to make discoveries and a name for yourself, and to work in an intellectually rich and not-that-badly paid environment. What’s not to like? Yet academics often seem discontented. Why? First, fear of failure. The freedom to make the calls on what to study comes at a price – the ever present risk of failure. Publishing in top papers and securing grants is a tough, competitive business. Most of us encounter difficult patches and some scholars start to panic or slowly sink and become disaffected. Second, success takes a long time to build – the delay between an idea and a paper in the New England Journal of Medicine is about five years on average, but can be as much as two decades. Gone is the immediate gratification that comes from a well-executed procedure or a skilfully conducted consultation, for example. Third, the administration in all universities is very tricky because academics and administrators are managing different risks – for the academic, the risk is failure to secure or deliver on a grant, while for administrators the risk is bureaucratic or legal challenge relating to contracting employment, finance, and so on. The more remote (centralised) the management, the worse the problem. My fourth reason is speculative and I hope it does not cause (too much) offense. Academics, being a non-random sample of the population, may be quite brittle people, tending towards introspective and somewhat narcissistic personality types – or maybe I am just describing myself! In any event, having tried the health and civil services, and lacking the courage to start a company, I plan to stay in a university for as long as I can cut the mustard.

— Richard Lilford, CLAHRC WM Director

Reference:

  1. Dellot B, & Reed H. Boosting the Living Standards of the Self-Employed. London: RSA Action and Research Centre. 2015.