Solicitors Qualifying Examination (SQE) is designed to have 2 stages (SQE stage 1, and SQE stage 2), of which SQE stage 1 is going to assess the candidates’ knowledge at point of admission. The method selected to achieve this objective is multiple-choice questions. What do we know about this type of testing?
By Dr Irene Rudique
Delivering SQE Stage 1
As explained by the Director of Solicitors Qualifying Examination, Dr Eileen Fry (Kaplan), SQE stage 1 will assess the candidates’ functional legal knowledge by presenting them with three papers, each containing 120 multiple-choice questions (MCQs) in such areas:
- Business Law and Practice, Dispute Resolution, Contract and Tort;
- Property Law and Practice, Wills and the Administration of Estates and Trusts, and Solicitors Accounts;
- Criminal Law and Practice, Public and Administrative Law and the Legal System of England and Wales.
SQE stage 1 is closely based on the QLTS MCT assessment. More information about what the MCT exams are can be found here.
According to Kaplan, the super-exam MCQs will examine the application of functional legal knowledge to realistic situations encountered by solicitors in practice. They will test key concepts and fundamental principles in each area, rather than isolated facts, memorised numbers or esoteric topics.
When writing bar exam questions, the designers of SQE stage 1 strive to avoid irrelevant difficulty by using concise and clear English and the wording that will not try to trick the candidates. The questions in the stem are expected to be focused and pose one specific task.
As for the answer options, the designers intend to keep them short, unambiguous, and to avoid vague terms, tricky wording or options which differ on more than one dimension.
Terms & Requirements
Before we proceed to the analysis of Kaplan MCQs, let’s define the key terms. A multiple-choice question presents a candidate with a question (called a stem) and multiple answer options associated with the stem. The options contain a single correct answer (called the key) and a set of incorrect answers (called distractors).
In order to be an effective method of assessing the testee’s knowledge, an MCQ test should possess four important characteristics of a good test:
- validity;
- reliability;
- objectivity;
- usability.
A test is valid if it can accurately measure a particular cognitive ability. Reliability refers to consistency of test results between two measurements. An objective test is free from personal error (i) in marking, on the part of the assessor, and (ii) in interpretation of test items, on the testee’s part. And finally, the usability of a test is determined by practical considerations – those of ease in preparation, administration, scoring and interpretation of test scores, as well as appropriateness of the time limit.
To make sure SQE stage 1 is valid, reliable, objective and usable, in November-December 2018 Kaplan conducted SQE 1 pilot recruitment and selected a representative group of 400 candidates to take the test on 20-22 March 2019.
Flaws & Benefits of Multiple-Choice Questions
Multiple-choice questions have a number of characteristics which make them a popular testing method. They are universally recognised as being easy to administer and quick to mark, which makes them relatively economical. When written well, a multiple-choice test can cover a wide range of material in a relatively short time.
At the same time, it is not economical from the viewpoint of preparation, as it is definitely cost- and time-consuming to create effective multiple-choice questions with appropriate distractors.
One of common objections against multiple-choice questions is that they only test memory without evaluating higher-order thinking skills. However, according to recent studies in a variety of fields (Applied Health Sciences, Business, Humanities, Mathematics & Science, Social Sciences), that is just the case with poorly written MCQs. At the same time, high-quality items ask examinees to evaluate situations, interpret facts, explain cause and effect, make inferences and predict results, thus evaluating different cognitive functions – not only memorisation, but also application and analysis.
Example of SQE Stage 1 Question
Now let’s proceed to the next stage and analyse a sample MC question provided by Kaplan:
STEM → A woman gives her second house (‘the property’) to her nephew. The property has never been the woman’s principal private residence and throughout her ownership the property has been used only by the woman and her family as a holiday home. Her nephew intends to use the property as his own principal private residence. The increase in value of the property during the woman’s period of ownership after deduction of allowable expenditure and the annual exemption is £5,000. The woman does not have any losses for capital gains tax (CGT) purposes.
Which of the following statements best explains whether CGT must be paid in respect of the £5000 gain?
DISTRACTOR → A. No, because the nephew intends to use the property as his principal private residence.
DISTRACTOR → B. No, because the disposal was by way of gift not sale and is therefore exempt from CGT.
DISTRACTOR → C. No, because the woman and her nephew can elect for hold-over relief to apply to the gain.
KEY → → → D. Yes, because the gift is a disposal for CGT purposes and no further exemptions or reliefs apply.
DISTRACTOR → E. Yes, because CGT is charged on all gains from disposals of land.
Discussion
As can be seen in the example above, the task opens with the stem which is made up of a short realistic scenario and a lead-in (question) based on it. Then follow five options, one of which is the keyed option and the four other are distractors.
The design of this test complies with major requirements to multiple-choice questions that have been developed in scientific literature.
- Based on empirical studies, the optimum number of properly functional distractors is 3 to 4, which means 4-5 answer options to a question.
- Another requirement is that the bulk of the reading must be done in the stem. Thus, answer options must be brief and no longer than the stem.
- To facilitate error-free processing, the stem must be a complete question or instruction ending respectively with a question mark or full stop. Besides, it is advisable to avoid negative questions in the stem, like ‘Which of the following statements does not explain…?’, as being under pressure, the examinees tend to overlook the not.
- The distractors must be well drafted and fully functional. That means they must run parallel to the key and be comparable. Researchers generally consider a distractor properly functional if it is selected by at least 5% of examinees. Answer options with a low difficulty index that are included in the test for humorous reasons, or to make students feel good about themselves, do not function as distractors thus compromising test quality.
- Well drafted answer options must not contain the ‘None of the above’ and ‘All of the above’ wording to avoid confusing or tricking the examinees.
The analysis of Kaplan’s sample multiple-choice question against the requirements above gives grounds to state that it fully complies with the guidelines agreed upon in the field and is a high-quality MC item that provides a valid measure of the candidate’s competence.
Stay with us and get e-mail updates on recent changes in the routes to qualification as a solicitor of England and Wales.
