Catherine Connolly (Galway West, Independent)
Link to this: Individually | In context | Oireachtas source

141. To ask the Minister for Education and Skills the details of the tendering process that resulted in Educational Testing Service getting the contract for the review of essential aspects of the coding used to implement the leaving certificate standardisation process; the value of the contract awarded to ETS; the number of companies applied for the tender; the way the tender was advertised; and if she will make a statement on the matter. [29228/20]

Catherine Connolly (Galway West, Independent)
Link to this: Individually | In context | Oireachtas source

142. To ask the Minister for Education and Skills the details of the tendering process for the contract, which was awarded to a company (details supplied) for the leaving certificate standardisation process in 2020; the number of companies applied for the tender; the details of the companies which applied for the tender; the reason given for the decision to give the contract to that company; and if she will make a statement on the matter. [29229/20]

Norma Foley (Kerry, Fianna Fail)
Link to this: Individually | In context | Oireachtas source

I propose to take Questions Nos. 141 and 142 together.

The decision to adopt a model of Calculated Grades by my Department was a direct result of COVID-19, which prevented the state from running the conventional Leaving Certificate Examinations.

As part of initial contingency planning for the Leaving Certificate 2020, the State Examinations Commission procured international expertise through Polymetrika International Inc.

Polymetrika was contracted to provide statistical and psychometric expertise, initially on the basis of contingency planning in the event of the Leaving Certificate not taking place as planned.

The value of that contract was €71,500 to cover 65 days. Any additional days are to be charged at €1,100 per day.

Polymetrika played an integral part of the development and running of the Calculated Grades model, and as such the contract for Polymetrika was extended to implement the Calculated Grades model under the pre-agreed terms of the contract. This was done following the decision of Government on 8 May 2020 to postpone the Leaving Certificate Examinations.

Due to the time constraints the Department availed of the procurement process known as the Negotiated Procedure without Prior Publication. This is used in circumstances where it is a case of extreme urgency.

To date, the total gross cost of services provided by Polymetrika under contract has been approximately €193,000 which reflects services provided to end September.

This expenditure reflects their work in both the contingency planning and the development and statistical work around the operation of the model.

As was announced on 30 September, the Department has also engaged the services of ETS, Educational Testing Service, a US-based non-profit organisation which specialises in educational measurement, to review essential aspects of the coding. ETS submitted their report of the review to the Department on 03 October.

The cost of services provided by ETS has not yet been finalised.

Catherine Connolly (Galway West, Independent)
Link to this: Individually | In context | Oireachtas source

143. To ask the Minister for Education and Skills the number of persons who have applied to sit the leaving certificate examinations in November 2020; the details of the closing date for applications for the November 2020 sitting; if, based on the current restriction levels nationally, she will state the likelihood of holding the exams in November 2020; and if she will make a statement on the matter. [29230/20]

Norma Foley (Kerry, Fianna Fail)
Link to this: Individually | In context | Oireachtas source

The deadline for registration for the postponed Leaving Certificate 2020 examinations is 5.00pm on Wednesday 07 October.

The State Examinations Commission has statutory responsibility for operational matters relating to the certificate examinations.

In view of this I have forwarded your query to the State Examinations Commission for direct reply to you.

Catherine Connolly (Galway West, Independent)
Link to this: Individually | In context | Oireachtas source

144. To ask the Minister for Education and Skills if the coding mistake in the leaving certificate standardisation process, which resulted in a student’s two worst non-core junior certificate grades being counted, rather than their two best non-core grades, was made by a company (details supplied) or if it was as a result of an error at Departmental level prior to instruction being given to; and if she will make a statement on the matter. [29231/20]

Norma Foley (Kerry, Fianna Fail)
Link to this: Individually | In context | Oireachtas source

The decision to adopt a model of Calculated Grades by my Department was a result of COVID-19, which prevented the State from running the conventional Leaving Certificate Examinations. The purpose of this process was to allow as many students as possible to progress to employment, further education and training, or higher education.

On 30 September, I made a statement advising that two errors had been identified in the Calculated Grades process. These meant that incorrect grades were issued to some students when they received their results on 7 September.

The first error was identified by Polymetrika International Inc as having taken place when processing students’ data. The error, in one out of 50,000 lines of code, was in relation to the processing of Junior Cycle data. The system was meant to draw on the core Junior Cycle subjects of English, Irish and Maths, and combine them with the students’ 2 best non-core subjects for all of the students in each class in each school. The coding error instead combined the core subjects with the students’ 2 weakest non-core subjects.

The error meant that, in some subjects, some students received Calculated Grades that were lower than they should have been, while some students received grades that were higher than they should have been.

In the course of a review which the Department then undertook, Departmental staff found a further error in the section of the code dealing with Junior Cycle results. The Junior Cycle subject Civil, Social and Political Education (CPSE) was meant to be disregarded as part of the model but was not. This second error, however, had a negligible impact on results.

I provided a further update on 03 October, announcing that improved Calculated Grades would issue to impacted students that day. This announcement was made following a review of essential aspects of the coding by Educational Testing Services, ETS, a non-profit organisation based in the United States.

ETS completed their review and provided it to my Department on 3 October. The review identified one further error, relating to how the code handled cases where students did not sit all three core subjects (Irish, English and Maths) at Junior Cycle level.

As a result of rectification of the errors, a total of 6,100 students have received higher grades. This breaks down as follows:

5,408 studentshave received a higher grade, by one grade band, in one subject,

621 studentshave received a higher grade, by one grade band, in two subjects,

71 studentshave received a higher grade, by one grade band, in three or moresubjects.

In addition to the students who received a lower grade than they should have in this year’s Leaving Certificate, some students received a higher grade than they should have. However, these students’ grades are not being amended and the students will not be affected.

ETS also identified an issue concerning how the algorithm used in the standardisation process treated students’ marks at the extreme ends of the marks scale (99% and over, 1% and below). However, ETS also stated that this had no material impact on the results and no student could receive a lower grade as a result of it.

A copy of the report submitted by ETS is available on .

Gary Gannon (Dublin Central, Social Democrats)
Link to this: Individually | In context | Oireachtas source

146. To ask the Minister for Education and Skills if the specification for the coding for the predictive grade model was signed off by her Department was explicit and clear as to the requirements; and the date on which this sign-off occurred. [29249/20]

Gary Gannon (Dublin Central, Social Democrats)
Link to this: Individually | In context | Oireachtas source

147. To ask the Minister for Education and Skills the person or body within her Department that approved sign-off for the predictive grade model; on what date, and at what project board meeting. [29250/20]

Gary Gannon (Dublin Central, Social Democrats)
Link to this: Individually | In context | Oireachtas source

148. To ask the Minister for Education and Skills the minutes of the meeting at which the specification and the subsequent approval were authorised for the calculated grade coding for a company (details supplied); and if she will provide evidence that the major flaws in the software quality assurance process will not be repeated. [29251/20]

Gary Gannon (Dublin Central, Social Democrats)
Link to this: Individually | In context | Oireachtas source

149. To ask the Minister for Education and Skills the professional standards that will be applied to future work on the ISO 29119 software testing standards which are an internationally agreed set which can be used by an organisation when performing any form of software testing; if not, the recognised standards that will be adhered to; and the specific name of the alternative standard. [29252/20]

Norma Foley (Kerry, Fianna Fail)
Link to this: Individually | In context | Oireachtas source

I propose to take Questions Nos. 146 to 149, inclusive, together.

Technical details of the Calculated Grades model and standardisation process were published on the date of issue of the Calculated Grades results and are available at .

The decision to adopt a model of Calculated Grades by my Department was a direct result of COVID-19, which prevented the state from running the conventional Leaving Certificate Examinations. A range of steps were taken to ensure adequate oversight of the Calculated Grades process.

The design of the Calculated Grades model was informed by advice from a the National Standardisation Group, comprising experts drawn from the State Examinations Commission, the Inspectorate of the Department of Education and Skills, the Educational Research Centre and international external expertise. The National Standardisation Group is the decision-making group responsible for the implementation of the iterative design and development cycles required to produce and refine the standardisation process and the application, review, and adjustment of the data in line with the commitments, principles, parameters and constraints associated with the calculated grades process to arrive at fair and just representations of student performance. The integrity, validity and reliability of the process of national standardisation was overseen by the National Standardisation Group.

The purpose, role and membership of the Group, along with information on the duration of its work and the decision making, governance and oversight arrangements that would apply, were set out in the paper Establishment of a National Standardisation Group for Calculated Grades.

The aim of the National Standardisation Group was to deliver a set of calculated grades that met the objectives of being fair and accurate at the point in the iterative process at which a safe, satisfactory and defensible set of outcomes has been achieved.

Details regarding this group, including its terms of reference, membership and its decision making framework are available in the Report from the National Standardisation Group to the Independent Steering Committee and the Programme Board which is available at.Minutes of the meetings of this group are available at

A non-statutory Independent Steering Committee to oversee the quality and independence of the process being undertaken by the Calculated Grades Executive Office was established by me. The Steering Committee comprises six independent members appointed by me and is chaired by Dr Áine Lawlor, former Director of the Teaching Council. The Steering Committee was established to provide assurance as to the quality and integrity of the outcomes of the Calculated Grades system. The 1st Letter of Opinion regarding the Calculated Grades System of the Independent Steering Committee can be found at: Details of the Independent Steering Committee including its membership are also available at the above link.

A Calculated Grades Programme Board provided governance and decision-making for the overall programme associated with the Calculated Grades Model and process. The Board was chaired by an Assistant Secretary General from the Department of Education and Skills and membership comprises officials from the Calculated Grades Executive Office, the wider Department of Education and Skills and the Department of Further and Higher Education, Research, Innovation and Science.

In addition, Dr Janet Brown, former Chief Executive of the Scottish Qualifications Authority, was appointed to the role of External Reviewer. The External Reviewer is an independent expert, unconnected with the design of the Calculated Grades model, who provides further independent oversight of the system and overall validation on the model, including the operation of the appeals system. Dr. Brown’s Statement in Advance of the Issue of Results can be found at .