A LINEAR PROGRAMMING BASED MODEL TO MEASURE EFFICIENCY AND EFFECTIVENESS OF UNDERGRADUATE PROGRAMS

Measuring performance of an educational program based on its academic achievement is not sufficient without considering the cost and the impact of the program. This paper aims to demonstrate the construction of a measurement model consisting of the input, output and outcome variables. The model can estimate both the efficiency and the effectiveness of undergraduate programs. After the aforementioned variables were identified for each individual efficiency and effectiveness model, a linear programming based tool, Data Envelopment Analysis (DEA) was used as the analysis method to integrate the models since it has the ability to consider all the variables simultaneously. The two models were integrated as a product, and was defined as the final model which was verified by applying it to measure the performance of 26 undergraduate programs in a university. The results show that seven programs are efficient, six programs are effective, and six programs are both efficient and effective. The model is flexible since it can be extended to include more variables or it can be modified by defining new variables in measuring efficiency and effectiveness of other programs or organizations.


INTRODUCTION
The success of an organization can be measured in terms of the efficiency and the effectiveness of the products or services offered by the organization (Hatry, 2006).Drucker (1977) stated efficiency as 'doing right things' and effectiveness as 'doing things right'.Efficiency relates the quantity of inputs used to produce outputs or services, whereas, effectiveness is measured in terms of whether the organization's goals have been achieved or not.In other words, effectiveness is about the relationship between input(s) and output(s) to the final objectives.The objective is the outcome(s), which is about the growth objective or the organizational ability to achieve its goals.From the perspective of public organizations, their goals are always associated with the welfare of the citizens.Goals and objectives can be achieved when institutions or organizations use limited resources to meet the stated objectives (Mancebon & Molinero, 2000).
In order to survive in a challenging world, it is essential for organizations that provide services to measure their performance as to whether they are able to meet the needs of their customers or not (Taylor & Godfrey, 2003).Many studies have been conducted on evaluating services from the perspectives of regular customers or users, or customers' service experts.For example, Ku-Mahamud and Othman (2010) interviewed travel experts to evaluate services provided by fourteen Asia Pacific airports by using the fuzzy subjective approach.Meanwhile, Adebayo (2008) requested cyber café users in Nigeria to determine factors that enhanced the performance of these cyber cafes, whereas Kasim, Ibrahim and Bataineh (2010) focused on the evaluation of the attributes of personal computers by university students in constructing computer preference index.In any case, meeting the customers' need is about the positive impact or outcome of the service.
In the context of Higher Education Institutions (HEIs), many services are offered to the society and the main ones are the undergraduate programs which are of diploma and first degree levels.In line with the national agenda of making Malaysian as the hub of higher education in the region, all Malaysia HEIs are monitored and measured continuously by the Malaysia Ministry of Education through various rating systems such as the SETARA Rating System which was first introduced in 2007(StudyMalaysia.com, 2012)).SETARA focuses on quality of the teaching, which is partially appraised by the students.In contrast, the Malaysia Research Assessment (MyRA®) instrument, developed by the Malaysia Ministry of Higher Education (MOHE) is for HEIs to selfassess the level of activities in research, development and commercialization (Kementerian Pengajian Tinggi Malaysia, Malaysia Research Assessment Instrument, 2011).Among the criteria evaluated in MyRA are the quality and quantity of research elements, post-graduates, innovations and networking.Currently, the SETARA and MyRA are the two main instruments used in measuring the performance of HEIs in Malaysia.However, these two existing measurement methods are based on single indicators of different dimensions and are more focused on the descriptions of quality.For example, Section D in the MyRA system focuses on 'Quantity of Postgraduates' and Section E is about 'Quality of Postgraduates'.For these sections, the universities are only required to supply separate information about the number of graduated postgraduates, the number of enrolled postgraduates and the ratio number of postgraduates to academic staff.Then, the overall performance of the university is usually computed by adding the individual values of the single indicators.
Therefore, the aim of this study was to construct a performance model that would help the HEIs to measure the efficiency and effectiveness of their undergraduate programs.The model consisted of input(s), output(s) and outcome(s) variables that were formed as the fractional linear programming model which used the Data Envelopment Analysis (DEA) tool as the analysis method, due to its powerful feature that could handle multiple variables simultaneously (Charnes, Cooper, Seiford & Zhu, 2004).The workability of the model was illustrated in measuring the efficiency and effectiveness of 26 undergraduate programs for 2009 in a public university in Malaysia.
The next section discusses the concepts of efficiency and effectiveness.It is followed with sections that provide a description of DEA, identification of input, output and outcome variables, the model, the results and discussions on the application of the model, and the conclusions.

CONCEPTS OF EFFICIENCY AND EFFECTIVENESS
Public performance measurement is a compilation of reports about the efficiency, quality and effectiveness of programs (Hatry, 2004).These measures are important for the public sector to improve its performance especially in terms of the provision of services.While, according to Mwita (2000), performance measurement is about the individual's or group's final output to achieve the stated goal and objectives.
The analysis of efficiency and effectiveness is often conducted based on the relationships between inputs, outputs and outcomes.The concept of efficiency was basically defined as a ratio of one output to one input (Farrell, 1957).Since that time, techniques to measure efficiency have improved and investigations of efficiency have become more frequent in both the private and public sectors (Charnes, Cooper & Rhodes, 1978).Nevertheless, the measurement of efficiency and effectiveness of public institutions remains a conceptual challenge.Problems arise because public spending has multiple objectives and public sector outputs are often not sold in the market.This implies that price data of the output is not available or cannot be quantified.Mandl, Dierx and Ilzkovitz (2008) suggested the conceptual framework of efficiency and effectiveness as illustrated in Figure 1, which makes a link between input, output and outcome.Effectiveness relates the input or the output to the final objectives or the outcome to be achieved.The outcome is often linked to welfare or growth objectives and therefore, may be influenced by multiple factors including environmental factors which may or may not be within the control of the policy-maker.For instance, if the efficiency of educational spending is analyzed closely, the wage structure may be seen as an external factor, whereas if the efficiency of the public sector is investigated as a whole, the wage structure might be an important input (Mandl, Dierx & Ilzkovitz, 2008).Furthermore, the distinction between output and outcome is often blurred (Afonso, Schuknect, & Tanzi, 2005).Output and outcome are used in an interchangeable manner, even if the importance of the distinction between both concepts is recognized.Therefore, the efficiency or effectiveness in any context, specifically in a higher educational institution context, is complex and often a controversial task.Applying efficiency or effectiveness to university performance measurement will inevitably involve the use of Furthermore, the distinction between output and outcome is often blurred (Afonso, Schuknect, & Tanzi, 2005).Output and outcome are used in an interchangeable manner, even if the importance of the distinction between both concepts is recognized.Therefore, the efficiency or effectiveness in any context, specifically in a higher educational institution context, is complex and often a controversial task.Applying efficiency or effectiveness to university performance measurement will inevitably involve the use of surrogate measures (Wilkinson, 1991).Therefore, a clear understanding of the conceptual difference in assessing inputs, university processes, outputs or outcomes is important.

DATA ENVELOPMENT ANALYSIS AND MEASUREMENT OF EFFICIENCY AND EFFECTIVENESS
Input(s) Output(s) Outcome(s) Environmental factors such as social economic back ground, climate surrogate measures (Wilkinson, 1991).Therefore, a clear understanding of the conceptual difference in assessing inputs, university processes, outputs or outcomes is important.

DATA ENVELOPMENT ANALYSIS AND MEASUREMENT OF EFFICIENCY AND EFFECTIVENESS
DEA is a linear programming model developed by Charnes, Cooper and Rhodes (1978) to produce the relative efficiency of each subject or the decision-making units (DMUs) under investigation.DEA is also a productivity measurement technique that measures the relative efficiency of public sector organizations which produce multiple outputs from multiple inputs in order to determine their performance.Efficiency measurement was found by Farrell (1957) but it only focused on the ratio of a single output to a single input.This is the drawback of the model because most organizations deal with multiple inputs and multiple outputs.Charnes et al. (1978) unfolds Farrell's model by introducing the DEA model with a different approach that has the ability to deal with multiple inputs and outputs simultaneously.Since then, there have been many DEA models that were derived to be used in measuring efficiency such as the Banker-Charnes-Cooper (BCC) model, the Additive Model, and the Charnes-Cooper-Rhodes (CCR) model (Charnes et al., 1981) but this study focused on the CCR model that assumes a constant return to scale.
DEA is also used extensively in measuring the efficiency of higher education institutions (Al-Bagoury, 2013;Ruzanita & Abdul Razak, 2002;Nazarko & Jones, 2014;Thanassoulis, Kortelainen, Johnes, & Johnes, 2010).However, very little research focuses on developing a mathematical model to measure the effectiveness dimension.For example, Sheth and Triantis (2003) used the fuzzy Goal DEA under the fuzzy environment but measured effectiveness as a deviational degree of achieved output to targeted output, whereas Hookana (2011) developed a measurement system for public institutions which included efficiency, effectiveness and quality dimensions through the Balanced Score Card Approach (Kaplan & Norton, 2001).

Inputs
Assessing the efficiency and effectiveness of public spending requires the measurement of the inputs entering into the production of public sector activities.This can be done in monetary and non-monetary (physical) terms.
Inputs are resources that influence the output or result obtained (Thanassoulis & Dunstan, 1994).In the higher educational institutions, academic staffs are the backbone of a university and the ranking of universities is often evaluated based on the number of professors that the universities have.So, since the undergraduate programs are the subjects to be evaluated, this study suggested lecturers from different levels of superiority as the inputs.Four types of academic positions were considered, namely, professors, associate professors, senior lecturers and lecturers.Tutors are excluded since they are nonpermanent staff or in temporary positions and have very little role in lecturing tasks.A professor is the highest position which an academic staff can achieve, followed by associate professor, senior lecturer and lecturer (Kasim, Kashim, Rahim & Hassan, 2015).

Outputs
The public sector, however, mostly provides non-market goods and services, which implies that their market value is usually unknown (Hatry, 2006) Output is the aim of the public sector to produce and it has to be defined.An option is to use a volume measure of outputs that allows efficiency or effectiveness to increase and decrease over time.In the context of universities, or specifically the undergraduate programs, the number of graduates or the degree attainments could be defined as the outputs (Kashim, Kasim, Khan, Rahim & Hassan, 2014).

Outcomes
The outcome has to be seen in a broader context which covers all the good longterm impacts of public programs and should capture the various dimensions of society values.Such achievements reflect the effectiveness of different kinds of programs or policy measures (Afonso & Aubyn, 2006).It is difficult to disentangle the effects of different outputs on the outcome (Afonso, 2005).Outcome is, moreover, often determined by external factors such as life style and socio-economic backgrounds.It is therefore, very difficult to isolate one transmission channel from another (Mandl, Dierx & Ilzkovitz, 2008).In the context of undergraduate programs the number of employed graduates could be defined as the short-term outcome, while the graduates' performance in their working atmosphere may be defined as the long-term outcome which is quite difficult to trace.This study defined the outcome of undergraduate programs as the number of graduates who were employed six months after their graduation.

THE INPUTS, OUTPUTS AND OUTCOMES IN THE CONTEXT OF HEIs
All the variables were identified through literature and the Malaysian Ministry of Education's documents.This study selected 'lecturers of different categories' as the inputs in order to produce 'graduated degree students'.It would not be accurate to consider all lecturers in one category because lecturers of different positions have different of tasks to accomplish and with different credibility.Furthermore, the tutors are not considered as the input since tutors are not permanent staff and they do not contribute directly toward the teaching activities in HEIs.It is not compulsory for tutors to teach and they usually go for study leave immediately after they are recruited by the university.Table 1 shows the identified variables to measure the efficiency of undergraduate programs, while Table 2 summarizes the variables to measure the effectiveness of undergraduate programs.

The Efficiency Model in the Context of HEI
The efficiency model for undergraduates to estimate the efficiency of 26 undergraduate programs in 2009 with four (4) inputs and one (1) output, and the efficiency of each program needs to be optimized one by one.The adopted model is as follows: (1) The undergraduate programs are said to be efficient if the score is one (1) but inefficient if the score is less than one.

The Effectiveness Model in the Context of HEI
Besides considering the efficiency model, this study implemented a model based on the conceptual framework by Mandl et al. (2008) subject to where where subject to where The undergraduate programs are said to be effective if the score is one (1) but ineffective if the score is less than one.

The Final Double Measure Model
The final model to measure both efficiency and effectiveness was basically integrated as the product of the two proposed models as follows.The overall performance of program e is (3)

RESULTS AND DISCUSSIONS
Table 3 shows the maximum, minimum, average and standard deviation values for the inputs: number of lecturers (L), number of senior lecturers (SL), number of associate professors (AP), and number of professors (P); outputs: number of graduate students (G); and outcome: number of employed graduates (EG) among the 26 selected undergraduate programs.The results of the evaluations as given in Table 4 were obtained by using equations ( 1), ( 2) and (3), and solved by the DEA-Solver software.The efficiency score and the corresponding ranking for each undergraduate program are given in columns two and three of Table 4 respectively, while columns four and five are the effectiveness scores and the matching rankings respectively.The final scores that represent the aggregated scores of efficiency and effectiveness scores for each undergraduate program are shown in column five with the corresponding rankings in column six of Table 4.The results reveal that only six programs are both efficient and effective and one program is efficient but not effective.Please note that a program is efficient if its efficiency score is one.Likewise, a program is effective if its effectiveness score is also one, and a program is both efficient and effective if both the efficiency and the effectiveness scores of that program are equal to one.As expected, the average efficiency score is higher than the average effectiveness score.This shows that the selected undergraduate programs are more efficient in producing graduates but less effective in getting graduate students to be employed.Furthermore, the average aggregated score is the lowest compared to the average efficiency score or the average effectiveness score.

CONCLUSIONS
This study has illustrated how a linear programming DEA-based performance model that measures both the efficiency and the effectiveness of undergraduate programs is constructed based on the framework proposed by Mandl et al. (2008).Here, the university, or specifically its main products, the undergraduate programs are treated as black boxes that produce outputs by utilizing the inputs and considering the impact of the outputs in society.The model if implemented can give the overall performance of the undergraduate programs and the university could improve more by taking proper actions based on the performance results.Measuring the efficiency of the undergraduate programs in producing graduates is not sufficient since the actual performance of a university is about how far it can contribute to the nation, specifically as a feeder to the workforce.In other words, both the efficiency and the effectiveness dimensions should be considered since a university is a place to produce potential brainy workers.

Figure 1 .
Figure 1.Conceptual framework of efficiency and effectiveness.
= effectiveness score of program   1 = number of employed graduates from program    = no. of input  utilized by program   1 = no. of employed graduates from program    = no. of input  utilized by program   1 = weight of number of employed graduates from program    = weight of input  from program = efficiency score of program   1 = no. of graduated degree students from program    = no. of input  utilized by program   1 = no. of graduated degree students from program    = no. of input  utilized by program   1 = weight of graduated degree students from program    = weight of input  utilized by program

Table 1
Variables to Measure Efficiency of Undergraduate Programs

Table 2
Variables to Measure Effectiveness of Undergraduate Programs to estimate the effectiveness of 26 undergraduate programs in 2009 with five (5) inputs and one (1) outcome, and the effectiveness of each program needs to be optimized one by one.The model is as follows: 1 = no. of graduated degree students from program    = no. of input  utilized by program   1 = no. of graduated degree students from program    = no. of input  utilized by program   1 = weight of graduated degree students from program    = weight of input  utilized by program

Table 3
Statistics of the Inputs, Outputs, and Outcomes  = effectiveness score of program   1 = number of employed graduates from program    = no. of input  utilized by program   1 = no. of employed graduates from program    = no. of input  utilized by program   1 = weight of number of employed graduates from program    = weight of input  from program    =   ×     = no. of input  utilized by program   1 = weight of graduated degree students from program    = weight of input  utilized by program = no. of input  utilized by program   1 = no. of employed graduates from program    = no. of input  utilized by program   1 = weight of number of employed graduates from program    = weight of input  from program    =   ×

Table 4
The Efficiency Scores, Effectiveness Scores and Aggregated Scores of Efficiency and Effectiveness