Search Results

This paper presents preliminary findings from a study investigating the application of a multi-source measurement approach to the assessment of higher order competencies in the public service industry. The aim of the study was to develop and validate a strategy to synthesise multiple sources of evidence to inform judgements of workplace competence. The methodology adopted integrates developments in two fields of study, performance appraisals and psychometrics. At present, 55 candidates have been assessed using a combination of assessment methods. This paper presents findings from a multi-faceted Rasch analysis of 360-degree assessment ratings. While candidate competence and item difficulty were well separated, there were only very small differences in the severity of different rater groups (self, supervisor, peer and subordinate). As such, rater group severity had no impact on candidate competence estimates.

Published abstract.

The collection of refereed papers is indexed at TD/TNC 73.622; individual refereed papers are indexed from TD/TNC 73.623 to TD/TNC 73.650. The collection of non refereed papers is indexed at TD/TNC 73.651; non-refereed papers are indexed from TD/TNC 73.652 to TD/TNC 73.680. Show less

The Victorian Qualifications Authority (VQA) developed a draft Credit Matrix Model for recognising the complexity of learning units delivered across the three educational sectors in Victoria, namely senior secondary education, vocational education and training [VET] and the higher education sectors. This paper reports on the empirical validation of the complexity component of the model, in particular, the appropriateness of the domains and indicators of learning thought to capture increasing complexity. To examine the validity and reliability of the draft Complexity Model, the indicators for each domain were empirically tested and validated through survey techniques. Participants were requested to (a) supply background information about the unit of learning; (b) judge which of the indicators within each measure best captured the complexity of their unit; and (c) rate each measure on its perceived effectiveness in defining the true complexity of the unit. Two hundred and fifty four (n=254) people responded to the survey. Of these people, 31 per cent represented senior secondary education, 44 per cent represented the VET sector and 25 per cent represented higher education. Item Response Modelling was then used to evaluate the psychometric properties of the measures as well as to empirically develop and define levels of complexity within and across the measures. The data collected demonstrated the validity, accuracy and internal consistency of each Complexity Measure. The results also showed that the correlations among the five measures were significant, yet moderately positive, thus having direct implications for the factor structure that should underpin the Complexity Framework. Two variations of the framework were produced from this initial investigation: a six level structure and an eight level structure, each underpinned by the five domains: Application, Autonomy, Accountability, Problem Solving and Knowledge.

Published abstract.

Available papers from the 8th Australian Vocational Education and Training Research Association (AVETRA) Conference, 'Emerging futures: recent, responsive and relevant research', are indexed from TD/TNC 81.564 to TD/TNC 81.610. Show less

The National Quality Council (NQC) funded a consortium led by Victoria University to design and implement 11 interactive sessions across each state and territory. The aim of the sessions was to communicate outcomes of the NQC's current work on assessment to schools involved in the delivery of [vocational education and training] VET qualifications recognised under the [Australian Quality Framework] AQF. This included schools that were either school-based [registered training organisations] RTOs and/or those schools who had partnership and/or auspicing arrangements with other RTOs. It was also expressed by the NQC that participation in the workshops was to be limited to school-based trainers and assessors delivering VET qualifications recognised under the AQF and not prevocational or similar programs. The relevant findings of the NQC funded projects to be communicated as part of the dissemination strategy included the reports and support materials available from the NQC website in relation to assessment, validation and moderation. The NQC completed these projects in 2009 and 2010 and was interested in ensuring that the key outcomes of the projects were shared with school teachers/assessors delivering nationally recognised VET qualifications

Excerpt from publication. Show less

To help achieve national consistency of assessment and reporting in the Australian vocational education and training sector, there is a need to develop a set of national principles for graded performance assessment. This paper challenges a number of prevailing principles from both a theoretical and assessment perspective, namely that grades must be criterion referenced (Williams & Bateman, 2003) [indexed at TD/TNC 71.04], meaningful (Rumsey, 2003) [indexed at TD/ANTA 75.21] and applied once competence has been achieved (Williams & Bateman, 2003). This paper argues that the use of generic criteria cannot be defended in terms of their validity and reliability and that a clear understanding of the underlying developmental continuum of learning is required to inform the development of meaningful and valid criteria and descriptors of quality performance. Finally, the paper proposes a set of principles that have been grounded in theory, have been put to the test in large-scale research, and are consistent with international literature on competence assessment.

Published abstract reprinted by permission of the copyright owner. Show less

From mid-1999 to the end of 2000, the Federal Government financed a Full Service Schools Program, one of three programs concerned with reducing youth unemployment by encouraging young people to complete Year 12 and make successful transitions to further training, new apprenticeships or work. At the same time, the Federal Government launched a program to identify ways of obtaining greater recognition by industry and university of achievement in VET (vocational education and training) courses at schools. This paper (a) focuses on the Business Services Training Package (BSB01), (b) analyses one aspect of a survey of the assessment of VET units in schools throughout Australia, and (c) compares Item Response Modelling (IRM; Rasch analyses) with a framework of rubrics that Subject Matter Experts had prepared to guide persons when they assessed Year 12 students. The rubrics were designed on the assumption that there was a continuum of competence within units of competence and that the continuum could be used as a basis for grading students for selection by universities. IRM showed that the Subject Matter Experts could make valid judgments of relative levels of difficulty of many criteria. In some units, some criteria with similar levels of difficulty could be combined. In other cases, the order of some of the criteria could be changed to consolidate the underlying continua of competence.

Published abstract reprinted by permission of AARE.

Selected papers from the 2006 AARE National Conference are indexed from TD/TNC 89.545 to TD/TNC 89.550 and from TD/TNC 89.701 to TD/TNC 89.710. Show less

This paper describes a study commissioned by the National Assessors and Workplace Trainers Body (NAWTB) to review existing Competency Standards for inclusion in a new and comprehensive training package for workplace training and assessment. The project team undertook a survey of user groups and the various state and national Industry Training Advisory Bodies (ITABs) as well as related specialist groups. A detailed outline of the survey results is presented, revealing that although the content of the current standards is regarded as generally satisfactory and relevant, with some reservations, the level of awareness and understanding of Competency Standards among users is extremely low, to the point where the national vocational training framework and the industry competence it underpins is endangered. The authors suggest that considerable revision of current competencies is required to meet Australian National Training Authority (ANTA) standards and that a reappraisal is needed of the existing vocational educational and training principles.

The volume of collected proceedings is indexed at TD/TNC 57.76; individual papers are indexed from TD/TNC 57.77 to TD/TNC 57.122. Show less

Maintaining the quality and comparability of qualifications has become a major priority for vocational education and training (VET) systems world-wide, especially for those countries which have decentralised assessment and reporting systems. In the absence of external examinations, one policy solution to align the assessment standards of different providers is the introduction of assessment moderation. Yet there has been little, if any, research conducted in VET that has examined different approaches to moderation for quality control purposes. By drawing on lessons learnt in a range of settings (i.e. schools, vocational and higher education), this paper identifies the optimal conditions for undertaking social, external and statistical moderation approaches to quality control. It also identifies a range of potential system level factors that could hinder implementation within the context of the Australian VET sector.

Published abstract reprinted by permission of the copyright owner. Show less

Despite systematic validation being an essential requirement within the National VET Regulator Standards for RTOs, numerous key stakeholders have continued to raise concerns with the quality and consistency of assessments being undertaken by [registered training organisations] RTOs for the purposes of issuing nationally recognised [vocational education and training] VET qualifications in Australia. In an attempt to improve industry confidence in such qualifications, six RTOs were commissioned, under the National VET E-learning Strategy, to engage industry employers and stakeholders to take part in the validation of assessments through the use of technology, referred to as e-validation.

This report first summarises the common features of the six e-validation programs piloted in terms of the unit/qualification focus, industry areas, partnership arrangements and assessment methods validated. It then attempts to summarise the common steps and activities undertaken by each RTO to prepare and conduct the e-validation. Next, the resourcing implications for the industry partners are discussed in terms of professional development needs, technology requirements, financial and workload implications. Finally, the benefits of the program and its implications for policy and practice are discussed.

Excerpts from publication. Show less

The purpose of this paper is to outline the role of quality assurance processes at both the national and regional level and also to outline the conceptual design for an East Asia Summit Technical and Vocational Education and Training Quality Assurance Framework (EAS TVET QAF). The EAS TVET QAF provides a set of principles, standards and quality indicators to assist EAS countries to assess the quality of their TVET systems and develop it accordingly, as well as provide a basis for greater alignment between national TVET systems. The EAS TVET QAF aims to address the needs of developing and developed economies in the region through an appropriate balance of compliance and evaluative approaches to quality assurance. It is flexible to respond to the varying needs and circumstances of countries. The EAS TVET QAF is capable of application at TVET accrediting and registering agency level and TVET training provider level, and it aims to bring benefits such as improved effectiveness, transparency and confidence in TVET provision within and across countries.

Excerpt from publication. Show less

Measuring literacy and numeracy proficiency and any changes in the level of skill are important in getting a sense of how well language, literacy and numeracy programs are working. The Adult Literacy and Life Skills (ALLS) survey and Australian Core Skills Framework (ACSF) are tools for assessing these. While the ALLS is a national survey conducted on an irregular basis, the ACSF can be applied at the individual level and provides evidence of progress. This report explores the two frameworks and finds that, while it is broadly possible to translate the five ACSF performance levels to the ALLS scales, there is not an exact match between the two, especially at the higher levels.

Published abstract. Show less