Does the Z-score method need improvement?
August 12, 2012, 7:33 pm,
the island
The Z-score method, has failed twice in ranking 2011 G.C.E. Advanced
Level (A/L) performances. Is this not an eye opener to the Z-score
method having some grave drawbacks?
Z-score Method
Since
year 2000, the Z-score has been used to rank G.C.E. (A/L) performances
for university admissions. It is considered a better scaling method
than the previous use of the aggregated marks for comparing student
performance in different subject combinations. However, the Z-score
method has come under an avalanche of public criticism since its
inception. Reaso ns for this criticism stem from the lack of
understanding and lack of transparency about the method. For an A/L
student, Z-score looks like a magic black box; it should not be the
case.
A student who sits the GCE (A/L) will get the grades
for subjects and the average Z-score for the three subjects sat as the
results of the examination. However, the unfortunate thing is that
there need not be a strong positive correlation between the grades and
the average Z-scores among different subject combinations. For
instance, a student with 3 ‘B’s in a subject combination might get a
better Z-score than a student with 3 As in another subject combination.
Thus, the innocent students get confused with their two sets of non
related results. But, when aggregate marks were being used as tool for
ranking it was not the case as there existed a linear relationship and a
strong positive correlation between the grades and aggregate marks. As
long as the raw marks are not used for ranking students, the grades
which are based on raw marks make no sense. On the other hand the above
grades cause unnecessary confusion too.
The Department of Examinations could consider one of the following suggestions as the means to allay unnecessary confusion:
1. It would be better to release the Z-scores for each subject rather than grades.
2.
Otherwise, the grades of the subjects should be based on Z-scores
rather than raw marks. For instance, for a particular subject, the
Grade ‘A’ can be given for a student who gets a Z-score of 1.0 or above
in the subject.
There is no perfect scaling method
available and Z-score is a widely accepted one. However, there are some
drawbacks in this method. Therefore, further research is needed to
find a better scaling method. Let us examine this in detail.
For
the calculation of Z-score, we do not need to assume any particular
probability distribution of the raw marks of a particular subject. The
formula for the Z-score is Z-score = (raw marks – measure of
location)/measure of dispersion; are mean and standard deviation are
being used as measures of location and dispersion respectively. Mean is
a good measure of location and standard deviation is a good measure of
dispersion for unimodal symmetric distributions. However, for
non-symmetric distributions mean is no longer a good measure of
location and standard deviation is not a good measure of dispersion
either. Therefore, we have to be careful in using Z-score for scaling,
when the raw marks follow any non-symmetric distribution.
In
order to have a unimodal symmetric distribution for particular
subject’s marks, the entire country has to be considered a homogeneous
population. Otherwise, there will be a possibility of having a
multimodal non-symmetric distribution. Still, we have district quotas
for university entrance and thus, we believe that all the districts are
not of the same standard. If so, how can we assume the countrywide
examination marks of a subject as a homogeneous population?
2011 A/L and Z-score
In
the year 2011, two different G.C.E. (A/L) examinations were conducted
for old and new syllabuses. While the repeat candidates sat the old
syllabus examination, fresh candidates sat the new one. Consequently,
for a particular subject, the Department of Examinations (DoE) had two
different sets of marks for the old and new syllabuses. Thus, when a
need for calculating the Z-score to rank and prepare lists of
candidates of both groups according to find a common cut-off point for
university admissions, the DoE found itself in a dilemma.
Earlier
the means and variances of the two different examination marks have
been pooled for the calculation of the Z-score of a particular subject.
However, Prof. R.O. Thattil, the person who introduced Z-score as a
tool for ranking A/L students in Sri Lanka, strongly opposed the above
pooling method. Later the Supreme Court’s verdict has also proved that
pooling is not an appropriate method.
Therefore it is clear
that if DoE wants to use the Z-score as a scaling method, it should
not pool the means and variances of the different examinations. If DoE
feels it appropriate to pool the means and variances of the different
examinations it should use some other scaling methods (not the Z-score)
for ranking purpose.
However, it is interesting to note
that Z-score calculations have become controversial even when they were
calculated separately following the court’s verdict. It seems there is
clear evidence that the repeat students were affected by this method.
In the recent past, on average 58% of the Medical seats were filled by
repeat candidates. But as per the 2011 separate Z-score results, only
26% (less than half of the past average) of the medical seats are being
filled by repeat candidates. Repeat students have been affected in the
engineering and management streams as well. It shows that the separate
Z-score is also not a good scaling method. However, note that pooling
is not a solution to this problem.
Why are the repeat Bio
science students affected heavily in the new (separate) Z-score
results? Since the historical data show that majority of the medical
seats were filled by repeat candidates, there could be two groups among
Bio science repeat students. One group wants to receive medical
education while the other wants to merely gain the A/L qualication.
Thus, there could be a high possibility that marks of the bio science
repeaters might follow a bimodal distribution. Thus the distribution
would not be symmetric and Z-Score method fails as a ranking method.
Median Centered Score
For
non-symmetric distributions, Median (which is the 50th percentile) is
the better measure of location, and Inter Quartile Deviation (IQD) is a
better measure of dispersion than standard deviation. Inter Quartile
deviation is the half of the difference between the 75th and 25th
percentiles.
We could define a new scaling method, Median
Centered Score (MCS), by MCS = (raw marks – median marks)/IQD of the
marks. The above MCS is not sensitive to extreme values, as median and
IQD are less sensitive to extreme values compared with mean and
standard deviation respectively. However, MCS is yet to be validated
using some real world data set. Moreover, further research is needed in
developing a scaling method for non-symmetric distributions.
Dr. S. Arivalzahan
Department of Mathematics and Statistics, University of Jaffna.
