Program Evaluation – How to Measure Learning Outcomes? Evaluation is the final component of the ADDIE (analysis, design, development, implementation and evaluation) educational design methodology. How do we measure the key performance outcomes of our degree program? (The CFEE’s Results box)

Gedeon (2017) “Measuring Student Transformation in Entrepreneurship Education Programs” identifies three Primary Goals of student learning: Knowledge (“The Head”), Skills (“The Hand”) and Attitudes (“The Heart”). Input Factor Goals that support these Primary Goals include faculty qualifications, student entrance requirements, resources available, amount of startup funding and vibrancy of the entrepreneurial eco-system. Output Goals include student satisfaction, showcase projects, awards, employment rate and number of startups.

“Assurances of learning [AoL] related to cognitive outcomes such as knowledge [Head] are best assessed using direct assessment methods such as tests and assignments (Martell, 2007). Assurances of learning related to behavioural outcomes such as skills and competencies [Hand] are best assessed using direct observation of performance or assignments (Palomba and Banta, 2001). Affective or attitudinal beliefs [Heart] are primarily assessed via indirect methods such as surveys, rating scales, and retrospective techniques.” (Gedeon, 2014 pp. 244)

Most large university entrepreneurship degree programs are accredited by either AACSB or EQUIS/EFMD. Although accreditation has been criticized as too prescriptive, too time consuming, too expensive and too focussed on Input Factor Goals, both provide good recommendations for measuring two of the three Primary Goals of student learning outcomes:

“The value add quality aspects of student transformation related to learning outcomes are particularly well quantified by both EQUIS (in Chapter 2) and AACSB (in Standards 16, 18, 19, and 21). Each program must identify specific student learning outcomes (e.g., communication skills and numeracy) and how they are measured. Each course within the program must then specify how these outcomes are taught and measured. Programs then track their students’ performance over time and identify continuous improvement through implementing new course content, pedagogies, or teaching methods. In this way, quality is measured as the value add or cumulative improvement in learning outcome assessments achieved by a student from start to finish of the curriculum [80].” (Gedeon, 2017 pp.5)

Gedeon & Valliere (2018) “Closing the Loop: Measuring Entrepreneurial Self-Efficacy to Assess Student Learning Outcomes” analyzes how accredited degree programs use tests and graded assignments to measure individual-level assurances of learning (AoL) related to knowledge (Head) and skills (Hand), and creates a new assessment tool for measuring attitudinal learning outcomes (Heart).

“All AACSB accredited degree programs must designate a set of learning outcome goals and define specific assignments and grading rubrics that measure whether or not students are achieving these learning outcomes (AACSB, 2016). The European Quality Improvement System (EQUIS) accreditation standards (European Foundation for Management Development, 2012) and the Association of MBAs follow essentially the same guiding philosophy, which arises from the TQM and CPI movements.”

“All these standards emphasize the need to measure direct output-based artifacts/documents that demonstrate that student learning has been achieved. This evidence-based assessment of learning thus requires a comprehensive process to set the learning outcomes and AoL goals, acquire the data, analyze the data, identify continuous program improvements, and implement changes (White, 2007). As there is no real consensus on how these learning outcomes should be selected, measured, analyzed, or used for feedback, each university must do this on its own (Sampson & Betters-Reed, 2008).” (Gedeon & Valliere, 2018 pp. 274)

Gedeon & Valliere (2018) perform a theoretically-based analysis of entrepreneurship degree learning outcomes to identify and define 8 specific skill-based learning outcomes (Hand), all of which are currently measured using AoL assignments with grading rubrics as part of the AACSB accreditation system of the Entrepreneurship Degree Program at Ryerson University.

But do students believe that they have actually gained these skills? Can we ‘Close the Loop’ to see if grades, feedback and demonstration of these skills change their attitudes (Heart) towards entrepreneurial self-efficacy (ESE)?

“Self-efficacy is defined as an individual’s belief in their ability to accomplish a job or specific set of tasks (Bandura, 1997)… The self-efficacy construct arises from social cognitive theory, which is based on human agency—people function as anticipative, purposive, and self-evaluating proactive regulators of their motivation and actions (Bandura, 2001).”

“When viewed from an educational assessment perspective, the self-efficacy construct is a robust predictor of learning and performance (e.g., Schwoerer, May, Hollensbe, & Mencl, 2005; Tracey, Hinkin, Tannenbaum, & Mathieu, 2001). Self-efficacy beliefs are strengthened through educational and training processes such as enactive mastery experiences, modeling, observational learning, and social persuasion (Wood & Bandura, 1989). Self-efficacy is also better than past performance (e.g. graded AoL assignments) as a predictor of future performance (Bandura, 1982).” (Gedeon & Valliere, 2018 pp. 276-7)

Gedeon & Valliere (2018) created a new 44 question survey to measure eleven different aspects of entrepreneurial self-efficacy (ESE). Potential ESE questions and scales from previous research associated with each of the 8 skill-based learning outcomes (Hand) were comprehensively reviewed and a new valid and reliable 44-item scale was operationalized and tested. It was discovered that there are 11 different factors or components of ESE (Heart) associated with these learning outcomes:

• Opportunity Identification and Creativity
• Information Alertness
• Planning Under Uncertainty
• Resource Marshaling
• Adaptability
• Financial Management
• Teamwork
• Critical Thinking
• Communication
• Persuading and Entrepreneurial Selling
• Self-Management

“The purpose of our research was to develop a valid and reliable ESE scale that may be used to augment a university’s AACSB or EQUIS accreditation system for an entrepreneurship degree program. To create this scale, we started not from a traditional intention model or theory of planned behavior foundation, nor a desire to differentiate non-entrepreneurs from entrepreneurs. We started from an educational assessment perspective and typical set of AoL learning outcomes within a TQM framework.”

“We ‘close the loop’ by not only measuring students’ demonstrated past performance [graded AoL assignments] but also understanding how students’ receive this feedback to alter their self-efficacy beliefs which have a powerful influence on future effort, motivation, and performance. Programs may use the data to identify program-level weaknesses to improve curriculum and pedagogy, as well as identify individual-level issues such as overconfidence and underconfidence to create meaningful student transformation and growth through self-efficacy beliefs.” (Gedeon & Valliere, 2018 pp. 294)

At Ryerson University we have been measuring and evaluating longitudinal data using this validated ESE instrument for 5 years on entrepreneurship degree program students, business students, and non-business students. We believe this is one of the world’s premier longitudinal datasets using multi-factor assessment of entrepreneurship degree program learning outcomes and effect of pedagogical techniques on entrepreneurial intent, ESE, core self-evaluation and desirability. We look forward to disseminating these results to entrepreneurship educators on the REI website when peer-reviewed publications become available.