https://www.e-assessment.com/glossary-of-terms/
Adaptive
Feedback
Feedback
given to students on an assessment item, where the feedback is modified
according to the student’s response , particularly where the response was
marked as incorrect, and typically aims to provide guidance on why the answer
was wrong.
Advanced
Question Types
Advanced
Question types are a level above Standard Question types. They typically
require complex setup and/or marking, or they contain complex marking
algorithms. Additionally, the level of interaction within the item may also be over
and above that of a Standard Question Type, and therefore they could be
considered as separate applications within a test.
Application
Programming Interface
The software
mechanism by which one computer program makes its functionality and data available
to another computer program. For example, a test-marking program might use an
API to request the correct marking key from the item bank program. APIs are
widely used in internet applications where programs are distributed across
different servers (for example in multi-vendor systems).
Assessment
An instrument
(e.g. an on-screen examination) used to make a judgement about learning, skills
acquisition or educational readiness or need against a set of pre-determined
criteria. Assessment instruments include examinations and tests, but also
portfolio and observation-based approaches to judgement.
Assessment
Delivery System
A combination
of software, hardware and communications components, as well as human processes
that manage the end-to-end process of administering e-assessments. Note: this
includes delivering the test package to centres, supporting the examination
administration functions, delivering the assessment to the candidate,
recovering their response, managing the marking and verification processes,
setting standards and providing assessment outcomes. It generally does not
include the writing and production of the assessment.
Assessment
Engine
The software
application that delivers an electronic test on a computer. The engine usually
consists of generic components (such as a web server) and proprietary software
(such as a specific e-assessment program). An e-assessment system typically
consists of an assessment engine, an item bank and an item and test creation
system.
Audio Capture
Question Type
The audio
question type enables the capture of sound created by the candidate. This could
be spoken sentences such as in a language test, or music from a performance.
Audio question types can sometimes start automatically as part of a question
workflow, or be controlled by the candidate as part of evidence capture. If
being used as part of language testing, it would be appropriate to combine the
audio capture question with either on-screen text or sound files. Visual
feedback on-screen is important so that the candidate is clear that the capture
has started and is recording audio. Capture can be open ended, however this
would generate larger file sizes. Consideration of file size is important, as
language testing could be being conducted in a location with poor internet or
power stability. Use of an on-screen avatar can sometimes help to improve the
quality of the conversation, if the candidate feels like they are speaking to a
real person, then the quality of the audio is less likely to suffer. Giving the
candidate the opportunity to playback the capture can also be useful. Audio
quality can be impacted by the confidence of the candidate, the quality of the
microphone or the background noise from other candidates. Simple techniques of
placing candidates completing an audio capture section of a test next to
candidate undertaking a written component can be good ways of reducing
background interference. The more practice a candidate has with this type of
question, the better they will perform. Classed as an Advanced Question Type,
audio capture questions are not typically auto-marked.
Authoring
Tool
Software used
to create items and tests for e-assessment. Note: the term is used more widely
to cover tools used for creating any content for delivery on-screen (e.g.
e-learning, general web pages, etc.).
Classical
Test Theory (CTT)
CTT is a set
of statistical measures relating to the performance of items within a test, and
the test itself, and used to provide evidence about the quality of a test and
the items within it. CTT’s most commonly used measures include facility (the
difficulty of a question), discrimination (the extent to which candidates’
performance on a question mirrors their performance on the test as a whole) and
Internal Reliability (the extent to which the test assesses a single
construct). CTT is relatively easy to implement and understand, but it cannot
distinguish between facets of the candidates and the questions (eg whether a
question is hard or a candidate cohort is weak), and for this reason Modern
Test Theory, including Latent Trait Theory, Rasch and Item Response Theory are
increasingly widely used.
Competency-Based
Assessment
An assessment
process based on the collection of evidence on which judgements are made
concerning progress towards satisfaction of fixed performance criteria which
describe the competency. Note: the competency-based assessment of an individual
takes no account of the performance of others in the wider group being assessed
(as is the case in norm-referenced assessment), and is typically limited to a
pass/fail grading (also called mastery/non-mastery). Competency testing is
typically used for licence to practice assessments (for example for doctors,
pilots, etc.).
Computer
Adaptive Test
A CBA test in
which successive items in the test are selected for presentation by a computer
algorithm drawing primarily on the properties and content of the items, and the
test taker’s response to previous items. Adaptive tests are typically used in
summative tests where a more accurate test result (compared to a linear test)
can be determined for a given test duration (or a shorter test can be
provided). For diagnostic purposes, adaptive tests allow a more detailed
exploration of strong and weak areas in a test taker’s knowledge within a given
time for a test. Two key features of adaptive tests are that (a) test takers
cannot return to questions once they have moved on and (b) the test taker’s
responses to items must be computer-marked.
Computer
Based Training
A term from
the 1980s describing learning which is mediated by a computer (typically a
stand-alone computer). Note: the modern term ‘e-learning’ differs in that there
is an expectation with e-learning that more of modern computing’s capabilities
(rich media, interactivity, etc.) will be used, as well as the computer’s
connection to internal networks and the internet.
Computer-Assisted
Assessment
Computer-aided
assessment (or “computer-assisted assessment”) describes assessments delivered
with the help of computers. This includes assessments delivered to the
candidate on-screen, developed on computer but delivered on paper, marked
on-screen or electronically (eg using OMR).
Computer-Based
Assessment
A subset of
CAA where the candidate is presented with the question on-screen and responds
to the question using the computer.
Content
Management System
A general web
term for a software tool which manages and maintains the technical and
editorial content of a website. Note: in an e-assessment setting, it describes
a software tool for managing assessment content (items and tests) and will
generally comprise an Authoring Tool/System and an Item Bank.
Delivery
Method
Delivery
method refers to the way in which the test is taken by the candidate, this can
be online, offline, secure locked down or open book.
Delivery
Platform
The platform
being used on the client device, for example PC running windows, Linux running
Ubuntu, Smartphone running Android, Apple Mac running iOS or windows etc.
Diagnostic
Testing
Non-accredited,
assessment used as part of a learning programme to identify a learner’s
strengths and weaknesses with a view to providing an appropriate learning
programme. Generally undertaken at the start of a programme, diagnostic
assessment therefore needs to evaluate learners’ existing levels of attainment
across the range of relevant knowledge, skills and understanding, so as to
inform personalised learning.
Digital
Logbooks
An online
record of practice experience and skills that are assessed. The logbook often
complements formal examinations.
Drag-And-Drop Question Type
The Drag and
Drop question type requires candidates to drag answer options into the relevant
drop zones; this can be useful for a number of question scenarios.Drag and drop
items can be text, video or image-based. Classed as a Standard Question type,
auto-marking is possible with this question.
e-Assessment
E-assessment
describes a range of activities where technology is used to enhance some or all
elements of an educational assessment process.
Common activities are on-screen tests (the candidate reads and answers
the question on-screen), electronic marking (the marker marks on screen –
either a scanned candidate script or an on-screen response), remote proctoring
(candidates taking on-screen tests are invigilated remotely), item banking
(test questions are stored in a database and assembled into tests electronically),
and e-portfolio (candidates assemble digital evidence of their learning for
skills assessments).
E-portfolio
An
e-portfolio contains digital items – ideas, evidence, reflections, feedback
etc., captured on the web which someone can present to a selected audience as
evidence of their learning and/or ability.
E-portfolio
Management system
An
e-portfolio assessment system enables the learning and skills of a number of
individuals to be captured and assessed online.
Either/Or
Question Type
The Either/Or
question type requires candidates to select the correct answer from two
different answer options and is most often used for questions where the options
are true/false or yes/no. Classed as a Standard Question type, auto-marking is
possible with this question.
End Point
Assessment
The final
assessment in the UK’s reformed apprenticeship, which takes place after the
student’s employer has signed off the student as ready for assessment.
Typically the assessment is graded and designed to show that the student is
competent across the range of skills and knowledge that have been covered
during their apprenticeship.
Equation
Entry Question Type
The Equation
Entry question type provides candidates with an equation creation tool, which
allows them to enter complex equations into the answer box. Classed as a
Standard Question type, auto-marking depends on the complexity of the equation
and the capabilities of the technology.
Essay Style
Question Type
An Essay
question requires the candidate to write an extended response in answer to a
question. Typically, this question type will provide the candidate with text
formatting features, and sometimes the ability to insert symbolic information
such as equations or symbols. Due the length of time a candidate may spend on
this question, it is important that their responses are securely stored at
regular intervals to prevent loss of data from due to computer error. Options
around the permitted response length can often be set to control the amount of
information the candidate provides. It is important that the candidate is able
to view all information provided either through an expanding window or
scrolling. Classed as a Standard Question type, auto- marking is not possible
with this question and human-marking is required.
Extended
Matching Question Type
The Extended
Matching question type allows item authors to create questions where candidates
are required to respond by making correct links between two lists of options.
There are a number of linking options available within this question type:
these include a one to one relationship, one to many, or many to many. Classed
as a Standard Question type, auto-marking is possible with this question.
Familiarisation
Test
A test used
by either the candidate or the centre. The candidate would use the test to
simulate the real assessment so that they can become familiar with the delivery
interface, question types and test structure. A centre may use the test to
highlight any technical or process problems ahead of the exam day.
File Attach
Question Type
The File
Attach question type allows test providers to present candidates with a file
which they can change or edit before uploading back into the test for marking.
This question type is perfect for any assessment that requires candidates to
use external software, such as Microsoft Word or Microsoft Excel. The files can
be pre-populated with information or can be left blank. The files can be
launched securely to prevent the candidate accessing additional files or other
applications. Typically both the candidate’s device and the marker’s device
will require a licence to the file type’s software, otherwise access to view or
edit the file might not be possible. Classed as a Standard Question type,
auto-marking is not normally possible with this question type and therefore
requires human-marking.
Fill in the
Blank Question Type
The Fill in
the Blank question type requires candidates to fill in blank spaces in a
passage of text. Multiple answer options can be set for multiple blank spaces
within a passage of text. Fill in the Blank can be used as an alternative to
using the Select from a List question type. The space provided for the response
typically expands to the answer given so as not to provide the candidate with a
clue to the response required. Classed as a Standard Question type,
auto-marking is possible with this question.
Hotspot
Question Type
A Hotspot
question requires the candidate to select one or more areas within an image.
Variations of this include providing a single point answer with options for
marking tolerance around the point. Multiple points can also be used along with
area mapping. In most cases it would be unreasonable to expect the candidate to
pick an exact pixel, therefore marking tolerance is normally recommended.
Classed as a Standard Question type, automarking is possible with this
question.
Interoperability
A feature of
computer systems components which allow the components to interact to exchange
information according to technical standards which define functionality useful
to the user. The IMS QTI specification is an example of an interoperability specification
within the e-assessment domain. IMS QTI allows the transfer of test questions
(and their associated metadata) from one system to another.
Item Response
Theory
IRT is a
commonly used Modern Test Theory statistical approach to measuring the performance
of candidates and test questions. Part of a wider group of Latent Trait Theory
approaches (including Rasch analysis) it is based on the idea that the
probability of a correct response to a question is a mathematical function of
both the candidate’s ability and the item’s difficulty (in contrast to CTT
where the difficulty of a question is fixed). IRT analysis is more complex to
implement and interpret but provides analysts with information to distinguish
between the ability of a candidate and the difficulty of an item, thereby
allowing comparisons of different candidate groups and/or different tests.
Likert Scale
Question Type
Typically
associated with surveys, the Likert Scale Question Type provides the candidate
with the means to answer according to a scale (for example “Strongly Agree,
Agree, Neutral, Disagree, Strongly Disagree”). These question types are
generally used in surveys. The scale may have fixed points and is therefore a
horizontal MCQ, or have variable points which would therefore need a degree of
marking tolerance. Classed as a Standard Question type, auto-marking is
possible with this question.
Locked Down
Locked Down
refers to the delivery method used during the assessment. An assessment in
Locked Down mode should prevent the candidate gaining access to applications or
internet sites not permitted during the assessment.
Manual
Test Generation
The manual
(human) process by which a test instance (a test form) is generated from a bank
of items (according to a formal or informal set of rules which may involve a
selection algorithm and randomisation). Note: this is widely used in
e-assessment settings where the candidates do not all take the test at the same
time, hence a variety of tests are required (to reduce the likelihood of test
and question exposure) but where known comparability is required between the
tests so that fair results can be given. Also see Automated Test Generation,
which is increasingly used where large numbers of test instances are required.
Metadata
Reference
data about a piece of information (e.g. an assessment item) that enables it to
be systematically stored in and retrieved from a database (e.g. an item bank)
according to a variety of selection criteria. In the context of assessment,
metadata might typically refer to aspects such as qualification or test
specifications, curriculum content and performance statistics. Note: metadata
is most useful when it conforms to an open standard, e.g. IMS LOM or QTI.
Mobile
Learning
A type of
e-learning where the learning is undertaken using a mobile ICT device (e.g. a
PDA, mobile phone or smartphone, handheld computer, etc.). The availability and
popularity of handheld portable devices has led to some research into the use
of m-learning techniques for e-assessment.
Multiple
Choice Question Type
A Multiple
Choice question (MCQ) gives candidates a number of answer options
(“distractors”) to choose from with only one correct answer (the “key”). These
are the most common on-screen question types in use within Computer-Based
Testing. Text is typically used in the answer options, but images or equations
may also be used. In some versions of MCQ tests, candidates may be asked to
provide their confidence level that the selected answer is correct, which can
then be used as part of the scoring calculation. Typically answer options are
randomised to reduce assessment malpractice, however care has to be taken not
to use punctuation on the last item only, or to use terms like ‘All of the
above’ if the intention is to randomise the answer order. Due to the correct
answer being presented on-screen, consideration is required when authoring to
prevent the correct answer from standing out, e.g. candidates can sometimes
guess at the ‘longest answer option’ using the logic that the author has had to
be explicit in the answer. This question type may also use weighted marking,
where the award for one option may be greater than another. Classed as a
Standard Question type, auto-marking is possible with this question.
Multiple
Response Question Type
A Multiple
Response question is similar to a Multiple Choice question, except more than
one answer option is correct and candidates may be asked to respond by
selecting all of the correct options. This type of question can sometimes use
images or equations. Variations may also include combination responses and
controls on how many responses can be given. Classed as a Standard Question
type, auto-marking is possible with this question.
Navigation
In an
e-assessment context, the on-screen buttons and other controls that move candidates
from screen to screen in an on-screen assessment (typically, from question to
question), and provide access to other non-question specific features such as
on-screen help, print functions, exit, etc). They are generally visually
separate from controls that relate to the specific question.
Non-question
Items
Non-question
items include: introduction pages, seen at the start of a test; information
pages, which may be seen by candidates during a test; and finish pages, seen by
the candidate after they have submitted completed their tests and submitted
their responses. Non-question items will not have a mark assigned to them.
Typically non-question items may be visible outside of a timed section of a
test.
Non-scored
Questions
A non-scored
item is a question (of any type) which has been included in a test to gather
performance data, but does not impact the score or result of the test for the
candidate. For example, it is a common practice to pilot new questions embedded
within a live test, but where the scores on the pilot items do not count toward
the candidate’s score.
Numeric
Entry Question Type
A Numerical
Entry question allows candidates to only enter numbers as the response to a
question. This can either be an exact value, or within a range set by the item
author. Consideration should be taken regarding any symbols which are expected
with the answer, such as $, £ or €. When considering international delivery the
use of commas or decimals should be considered. Classed as a Standard Question
type, auto-marking is possible with this question.
Objective
Question
A test item
in which the response is evaluated against objective criteria. This might be a
single simple response, such as in a multiple choice item where the criterion
is whether the student has made a correct selection. It might be that the
student’s response has certain properties which can be established objectively.
In CAA this is usually done automatically.
Offline
Assessment
An on-screen
assessment which is conducted without using an internet connection during the
test (although an internet connection may well be used to deliver the test to
the client computer prior to the test starting, and to upload the candidate
responses once the test has completed).
On programme
assessment
Assessment in
the UK’s reformed apprenticeship, which takes place during the apprenticeship
learning programme, and prior to the student being signed off by the employer
for the End Point Assessment
On-Demand Assessment
Used in
public examinations. Assessments where there is a high degree of flexibility in
the date and time that tests can be offered to suit the student or their
learning programme (although it may not necessarily include all days, times and
dates). In contrast to many traditional assessments which are provided on a
fixed date and time (or a limited range of dates and times).
On-Screen
Assessment
An assessment
delivered to the candidate on a computer screen, and where the candidate
provides their response on-screen (for example by typing, or clicking on the
correct response).
Online
Assessment
An on-screen
assessment which relies on an internet connection during the test to download
subsequent questions and upload candidate responses. Sometimes termed
“conducting a test live over the internet”.
Online/Offline assessment application
The ability
to capture and assess evidence of learning and skills to an e-portfolio when àn
internet connection is unavailable and for this evidence and assessments to be
added to the e-portfolio once an internet connection is established.
Open source
e-portfolio
This is a
personal learning environment mixed with social networking, that allows an
individual to collect, reflect on and share their achievements and development
online in a space they control.
Open
Standards
Shared,
freely available and internationally agreed standards for computer-based
systems, designed to enable communication and interoperability.
Optical
Character Recognition
Optical
Character Recognition (OCR) is a means by which a computer can recognise text
and other marks in handwritten responses on paper that have been scanned, and
convert these to digital format. OCR is often used in assessment to
electronically mark paper responses to multiple choice tests (for example, on
bubble sheets)
Optical Mark
Reader
A device that
scans paper-based tests and converts marks made by the student using pen or
pencil into digital data.
Parameterised
Item Type
An item where
parts of the question are generated according to a formula embedded within the
question. This randomisation is usually undertaken dynamically, using the
formulae, as the test is generated or delivered to the student, in contrast to
cloning where item variants are generated during authoring by use of the
randomisation parameters. The term parameter refers to the variables that are
used by the formula to create the item instances. Many different types of
questions (e.g. MCQ, gap fill, etc.) may use parameterisation.
Personalisation
The
configuring of an IT system by students to suit their personal requirements
(e.g. selecting preferred font sizes and colours, volume levels for audio, et
al). Also refers to more complex customisations of the user experience to meet
personal learning needs.
Polytomous
Item
An item
having more than two response categories. For example, a 5-point Lykert type
scale where items can be scored 0, 1, 2, 3, 4, or a partial credit question
where candidates can score between 0 and 4 marks. Polytomous is a term
typically associated with IRT and other forms of latent trait analysis.
Portfolio
Assessment
An assessment
where a student’s portfolio of assembled work is assessed. This type of
assessment is distinct from a test (which is administered on a single occasion)
Practicability
The
feasibility of an assessment in terms of operational efficiency and viability.
A valid and/or reliable assessment may not be practical due to the cost or time
required to carry it out. High quality assessments are valid, reliable and
practicable, although there are typically trade-offs to be made between these
three key elements of test quality.
Proprietary
Software
Software
requiring a licence (for which a charge is usually made) from a particular
company.
Psychometric
Test
Provides one
or more measures of a candidate’s personality traits. It is most commonly used
as part of the selection process for employment, or to support careers
guidance. Typically it seeks to place candidates on a number of scales
according to their preferred behaviours or aptitudes for aspects such as
working with others, managing pressure, preferred working environment and
thinking style.
QTI Lite
A
simpler-to-implement version of the QTI technical interoperability
specification for tests and items which allows tests developed in one system to
be delivered to candidates on other systems (currently at version 1.2). See
www.imsglobal.org/question/index.cfm#version1.2lite
Question and Test Interoperability
A technical
specification for tests and items which allows tests and test items to be
authored and delivered on multiple systems interchangeably. It specifically
relates to content providers (that is, question and test authors and
publishers), developers of authoring and content management tools, assessment
delivery systems and learning systems. It is designed to facilitate
interoperability of assessment content between systems.
Rich Feedback
Feedback
which goes beyond providing the correct or model answer to an item, and a
simple explanation of why the student’s selected response was wrong. Rich
feedback is usually personalised to the candidate’s response and designed to
deal with the underlying misconception.
Select from a List Question Type
The Select
from a List question type requires candidates to select missing word(s) from a
passage of text by choosing from a drop-down list of answer options. Select
from a List items can often be used multiple times within a passage of text.
Random ordering of the answer options may be enabled to help prevent
malpractice opportunities during the assessment. Classed as a Standard Question
type, auto-marking is possible with this question.
Short Answer Question Type
A Short
Answer question will allow candidates to respond to a question with a word or
short phrase, as specified by the test author. Multiple answer variations can
be provided and settings such as case sensitive can be included. Classed as a
Standard Question type, auto-marking is possible with this question, although
some responses may need to be human-marked.
Source Materials
Source
materials are files such as images, PDFs or editable material that are made
available to a candidate during an assessment (for example containing reading
material, formulae, etc.). It is useful for the candidate to be able to
interact with this material by making notes or highlighting sections.
Spreadsheet
Question
The
Spreadsheet question type is common in assessments of accountancy. By embedding
a spreadsheet within a question, the candidate does not need to access a third
party application. The question can be pre-populated with data to be statically
visible to the candidate or enabled for editing by the candidate. The question
can also be provided to the candidate as blank so that they may present their
data in a form deemed suitable. The spreadsheet question type allows the
formatting of text and data, and also supports the inclusion of formulae and
other common spreadsheet features. The spreadsheet question may also provide
candidates with the ability to justify their responses with annotation. Classed
as an Advanced Question type, auto-marking can be achieved by logic-based
marking rules which can mark a fully correct answer or an answer where the
calculation is correct however the source data is incorrect.
Standard
Question Types
Standard question
types are typically QTI supported items; however, definitions vary. In some
locations, standard question types like Drag and Drop can be considered
Advanced. A preferred way to define standard question types is where the
authoring and marking do not require complex coding, and that the item can
typically be auto-marked by the computer, or marked by a human without the need
for complex marking tools.
Table
Question
Tables are
commonly used for the presentation of data or to provide a means by which a
candidate can record their responses in table form. The table question type can
either be created by the test author and then presented as static information
for the candidate, or provided as a partially completed table for editing by
the candidate. The question type could also be provided as a response option
alongside other options such as graphs and charts. The table question may also
feature formatting of text and alignment of cells. Classed as a Standard
Question type, automarking is possible with this question type.
Test Package
The
electronic package of files in an eassessment system which includes the test
content as well as any embedded applications, resources, and sometimes the test
player, which is delivered to the Client PC for the student to undertake. The
test package may have travelled from a central server at the Awarding
Body/Certification Agency, to a local server in the test centre and then to the
Client PC, and will be subject to considerable file and channel security in the
case of a high stakes assessment.
Test Player
or Test Driver
A piece of
software which resides on the client PC and “runs” the test content package,
i.e. displays the questions on-screen, collects the student responses, controls
the exam clock, and access to other resources, etc. Some test players are
considerably more complex than others. For example, a test player for MCQ items
may be little more than some scripts embedded in standard HTML pages, whereas
the test player for innovative items or a sophisticated test made up of
simulations and embedded applications would be a substantial computer
application in itself. Sometimes test players are pre-installed on client PC’s
and sometimes they are delivered to the client PC along with the test content
itself.
Video Capture
Question Type
The video
capture question type enables the capture of audio and video. Associated more
with evidence-based assessment, files can either be recorded directly into the
assessment or attached as evidence. Classed as an Advanced Question Type,
marking would typically be undertaken on-screen by a human marker.
Web-Based
Assessment
An assessment
delivered from a server via the Internet or an Intranet (such as a centre or
education authority Intranet) and where candidates access the assessments using
a standard browser. If Internet-based, the assessments could be online with
candidate responses delivered to the server in real time for automatic marking
and immediate feedback.
