Efficiency vs effectiveness: an analysis of tertiary education across Europe

Ozana Nadoveza Jelić*

Ozana Nadoveza Jelić

Affiliation: University of Zagreb, Faculty of Economics and Business, Zagreb, Croatia

0000-0002-3651-7795

Correspondence
onadoveza@efzg.hr

   

Margareta Gardijan Kedžo*

Margareta Gardijan Kedžo

Affiliation: University of Zagreb, Faculty of Economics and Business, Zagreb, Croatia

0000-0001-5390-057X

Article   |   Year:  2018   |   Pages:  381 - 414   |   Volume:  42   |   Issue:  4
Received:  June 1, 2018   |   Accepted:  October 21, 2018   |   Published online:  December 14, 2018

Download citation

https://doi.org/10.3326/pse.42.4.2

Abstract

1 Introduction

2015

2013

2015

2000

2001

2006

2014

2013

2013

2001

Due to the importance of educational services for growth, attitudes and political and social awareness, they are provided and publicly financed, to a greater or lesser extent, by practically all governments around the world. Additionally, educational externalities are a textbook example of market failure and one of the most important motives behind government intervention in this sector. According to Szirmai (2015), after World War II, expansion and improvement of education were generally considered essential to development. The awareness about the role of education in the development process resulted in a far reaching education expansion. Over the course of time, increased government expenditures on education translated into higher levels of education. Consequently, higher education enrolments have grown significantly over the last three decades. According to World Bank (2018) data, the world gross enrolment ratio in tertiary education¹ grew from 13% to 35% during the 1985-2015 period. Growth has been even more impressive in the European Union (EU) where the average annual growth rate of the gross enrolment ratio in tertiary education reached 3.5%. This has led to an increase in the gross enrolment ratio in tertiary education from 25% in 1985 to 68% in 2014.

However, as Szirmai (2015) puts it “Since the 1970s, optimism about the contributions of education has been shaken and more emphasis is given to improving the quality of education.” This author notices that not all educational investments are effective and efficient in the development process. Due to the potential ineffectiveness of educational inputs, the quality of education can be unsatisfactory. Thus, the rising educational coverage and duration of education, as well as government and even private educational expenditures, are not always efficiently transmitted into higher productivity and wages, growth rates and better institutions. Therefore, it can be argued, it is not quantity that underlies the successful exploitation of all forms of educational benefits, but the quality and the effectiveness of the educational inputs and investments. Although efficiency and effectiveness are similar concepts, they are not synonyms. Viljoen (cited in Kenny, 2008) defined efficiency as relating to “how well an activity or operation is performed.” The term effectiveness relates to performing the correct activity or operation. In other words, “efficiency measures how well an organization does what it does, but effectiveness raises value questions about what the organization should be doing in the first place”.

Conclusions of various DEA-based studies sometimes differ significantly, which makes it impossible to draw general conclusions concerning tertiary educational efficiency at the EU level. Differences in conclusions mostly arise from the diverse selection of inputs and outputs considered within different studies. Additionally, some papers deal with a narrow sample of countries (e.g. Ahec Šonje, DeskarŠkrbić and Šonje, 2018; Yotova and Stefanova, 2017), i.e. homogenous countries with similar development levels, and others deal with a broader and more or less heterogeneous set of countries, which can also affect the difference in the results (Aubyn et al., 2009; Aristovnik and Obadić, 2011; and Toth, 2009).

2 Literature overview

2018

Yotova and Stefanova (2017) used the same method on a set of countries similar to that chosen by Ahec Šonje, Deskar-Škrbić and Šonje (2018). As an input variable, authors used total expenditures on tertiary education per student as a percentage of per capita GDP in 2012, while the set of educational outputs variables included three indicators: tertiary educational attainment (age 25-34), the employment rate of the population with tertiary education outside the risk of poverty and social exclusion and the mean monthly earnings of a person with tertiary education as a share in per capita GDP in 2014. Again, the analysis is limited to one input and one output. It should be noted that both studies include some educational output quality indicators, but they do not consider any educational input quality measures, which could lead to biased results and conclusions.

Toth (2009) analyzed the efficiency of tertiary education in 20 EU countries in 2006 using output-oriented DEA with variable returns to scale (VRS). The author used a ratio of expenditures spent on higher education to GDP as an educational input, and the ratio of people with a degree to the total population as well as the employment rate of people with a degree as educational output variables. Beside standard outputs and inputs, the author used two non-discretionary variables (parental educational attainment and public-to-total expenditure GDP per capita in current US$). However, Toth’s (2009) results differ significantly from other, related, studies that include EU countries³. She found that, for example, Denmark and Italy (among others) share the first position regarding tertiary education efficiency in 20 analyzed EU countries, while Aristovnik and Obadić (2011) and Aubyn et al. (2009) rank these countries as relatively inefficient.

Aristovnik and Obadić (2011) used output oriented DEA with variable returns to scale (VRS) to assess tertiary education efficiency in a broad set of countries (selected group of EU and OECD countries) during the 1999-2007 period. The analysis included input data on expenditure per student (tertiary, % of GDP per capita), school enrolment (tertiary, % gross), and output/outcome data, i.e. school enrolment (tertiary, % gross), labor force with a tertiary education (% of total) and the unemployed with a tertiary education (% of total unemployment). To assess technical efficiency regarding different inputs and outputs/outcome, the authors tested three. Two out of three considered outputs are standard educational quantity output indicators, while the last can be regarded as a quality indicator. In the conclusion authors emphasize the need to consider some educational quality data.

The most comprehensive study employing DEA methodology to assess the efficiency of the tertiary education in a broad set of countries is authored by Aubyn et al. (2009). The authors used two approaches: input and output-oriented DEA with variable returns to scale (VRS). The analysis is conducted over two subperiods: 1998-2001 and 2002-2005. In the first model, authors used a number of academic staff and students as inputs, while the second model considered spending in private government-dependent institutions (in % of GDP) as an input variable. A weighted number of graduates and a weighted number of published articles were used as output variables in both models. All educational inputs and outputs considered in this paper can be regarded as quantitative. However, the study includes a number of non-discretionary measures such as selection of students, budget autonomy, staff policy, output flexibility, evaluation, funding rules and PISA results⁴, which can be seen as qualitative measures (mostly) of inputs. 

It should be noted that conclusions differ in the abovementioned papers, which makes it impossible for us to draw any general conclusions on tertiary educational efficiency at EU level⁵. We suspect that differences in conclusions mostly arise from the diverse selection of inputs and outputs considered within different papers. However, the differences in the conclusions of the reviewed papers also arise because of the different samples of countries. That is, two papers deal with a narrow sample of countries, i.e. homogenous countries with similar development levels, and others deal with a broader and heterogeneous set of countries, which can also produce different results. Still, differences arise even if the samples are relatively similar. For example, Aristovnik and Obadić (2011), and Aubyn et al. (2009) use the same number and coverage of countries and even time periods in different model specifications (variables), but sometimes the results differ significantly. For example, the first model in Aristovnik and Obadić (2011) ranks the Czech Republic as the first and then as the 33^rd in the second model. Similarly, in Aubyn et al. (2009) Cyprus is ranked number one in the first model (1998-2001) and then as 27^th in the second model (1998-2001)⁶.

3 Data: tertiary education inputs and outputs

2006

To our knowledge, there is no precise definition and delimitation of quantitative and qualitative educational inputs and outputs. According to Lee in Bourguignon, Elkana and Pleskovic (2007), an outcome of education is composed of both the quantity and the quality of educational capital. According to him, the quantity of educational capital can be measured by the number of graduates. However, he emphasizes that it is rather difficult to measure the quality of education accurately. The author adds that the quality of education is reflected in the performance of students and graduates, as the value added of schooling can be measured by labor market performance, such as extra earnings or employment, of educated workers. Due to the lack of official quantity vs quality definitions regarding educational inputs and outputs, in this section, we provide the basic rationale behind the choices made in this paper.

Before the provision of details regarding the selected inputs and outputs, figure 1 gives a synthetic overview of educational inputs and outcomes, as defined in Scheerens, Luyten and van Ravens (2011).

3.1 Quantitative measures of educational inputs

3.2 Qualitative measure of educational inputs

3.3 Quantitative measures of educational outputs

3.4 Qualitative measure of educational outputs

An average overall score of Times Higher Education university rankings is chosen as an output indicator of the tertiary education quality in the last sub-period (2013-2015). We considered other ranking lists, but Times Higher Education was the only university rankings database which covered all countries in our sample in 2016. In previous sub-periods (2004-2012), we used the gross domestic product in PPS per capita (% of average) as a proxy for tertiary education outputs quality due to the incompleteness of the university rankings data and their correlation with university rankings (overall score). Anecdotal evidence presented in figure 2 justifies this choice. Namely, it seems that the correlation between the GDP per capita and the average university overall score (measure of the educational outcomes quality) using the ranking of the Times Higher Education (2017), significantly exceeds the correlation between the GDP per capita and the tertiary educated population as a percentage of 15-64 years aged population (typical measure of educational outcomes quantity).

4 Methodology

The efficiency and (what we later regard as) the effectiveness analysis of the tertiary education in 24 EU member states⁸ is conducted using data envelopment analysis (DEA). DEA is a nonparametric method of mathematical programming, which is developed for evaluating the relative efficiency of units under assessment, usually called the decision-making units (DMUs). Since its introduction by the pioneering CCR model in 1978 (Charnes, Cooper and Rhodes, 1987), followed by the BCC model published by Banker, Charnes and Cooper in 1984, DEA has instantly been recognized as a modern tool for performance management. While the CCR model assumes constant returns to scale (CRS), the BCC model assumes variable returns to scale, which allows the use of DEA in problems where increases in inputs result in non-proportionate increases in outputs (and vice versa). The most appealing features of DEA are that it allows multiple criteria for determining efficiency to be used and appropriate variables to be selected, which are (in most models) unit-invariant, without the use of their pre-defined weights. In addition, all assessments are relative given the finite number of comparable DMUs. Following the specific needs of the research environment, a vast number of models have been developed within DEA to fit and capture the nature of the research problem, thus providing a great tool for different kinds of efficiency analysis. Additionally, the popularity of DEA and the number of its applications are on the rise (Emrouznejad and Yang, 2018).

DEA was initially developed with the idea of measuring the efficiency of production units, such as factories, hospitals or banks, where one can unswervingly determine their inputs and their outputs. Such DMUs can manage their inputs and outputs to a certain degree (thus the name decision-making units). An additional assumption is that the aim of DMUs is to use their available inputs to achieve greater outputs or try to use fewer inputs for producing the desired level of output. In other words, they are assumed to aim for the efficiency in a production process. However, the application of DEA has spread outside the production processes and researchers are using it for evaluating the relative efficiency of different kinds of (relatively homogenous) units that need to be estimated given their undesirable (input) and desirable (output) characteristics. The examples are the portfolio selection, the performance of companies using their financial ratio data, performance of countries according to their macroeconomic indicators or different “processes”, for example, fiscal policy or educational policy. As is obvious, such DMUs are not the “decision-making” units themselves and not all of them should aim for efficiency in terms of fewer inputs to greater outputs. Moreover, the selection of their inputs and outputs is arbitrary, but this allows a researcher to define the relevant aspects of the “efficiency” of DMUs.

The use of DEA for estimating the relative efficiency of education at different levels (primary, secondary, tertiary) has been very popular over recent years. The overview of some of these researches, previously mentioned in the literature overview, revealed that the most frequently used model is the BCC model (with input or output orientation), which is an appropriate approach given the nature of this research problem. Without questioning the great contribution and effort of past researches, what we argue is that their selection of inputs and outputs gives more importance to the greater quantity of the educational output. We strongly suggest that education should be assessed not only in terms of quantity but also in terms of quality. Figuratively speaking, a factory that manages to produce something using almost nothing should be seen as a role model, and a factory that invests a lot relative to others and achieves less than the others should be recognized as poorly managed. However, countries that have large investments in education should not be punished in such studies if they manage to provide a high quality of education. Likewise, the countries that have almost negligible inputs should not be rewarded just because they managed “to produce” any amount of outputs of low quality despite their low inputs. Therefore, we suggest that at the beginning of the study using DEA, the crucial question should be asked: “Are we really aiming at the quantity or the quality?” and the answer should be followed with the selection of the inputs and the outputs that are relevant for the study. 

In addition, just as the output of the production facility is determined with the quality of the inputs, which cannot be always controlled, certain levels of the educational process are determined by the outputs of the preceding processes. Figuratively, one cannot make a tasty cake using salt instead of sugar. For this problem, DEA allows the definition of non-discretionary inputs, which are relevant but they are not controllable and are defined by the environment (Banker and Morey, 1986). This approach was used in some previous studies of education using DEA. However, as we will explain in the following paragraphs, we will treat the non-discretionary variables as discretionary to provide results that are more informative. 

DEA models can be output oriented, aiming at maximization of outputs for the given level of inputs, input-oriented, aiming at minimization of inputs for the given level of outputs, or non-oriented. Also, the models can assume constant, variable or generalized returns to scale. Following the nature of the problem we are analyzing, we decide to use the output-oriented model assuming variable returns to scale (BCC model). To explain the methodology, we first formulate the model. Let there be N decision-making units (DMUs): DMU₁, DMU₂,…, DMU_N which are homogenous and comparative. We assume that their efficiency should be estimated in terms of a certain number of inputs – the variables the values of which we want to be as small as possible, and a certain number of outputs – variables the values of which we prefer to be as big as possible. Let x_ij ≥ 0 be an i-th input for some DMU_j, i ∈ {1,…, m} and y_rj > 0 its r-th output, r ∈ {1,…, s}, j ∈ {1,…, N}. Therefore, each DMU_j is represented by a vector of inputs x_j = (x_1j, x_2j,..., x_mj) and a vector of outputs y_i = (y_1j, y_2j,..., y_sj), so X = [x_ij] ∈ R^mxN is an input matrix and Y = [y_rj] ∈ R^sxN is an output matrix. To make the model stable, it is recommended that the number of DMUs (N) should not exceed max {ms,3(m+s)}. The BCC model (Banker, Charnes and Cooper, 1984) can be written in the following envelopment form:

(1)

(2)

(3)

(4)

Xij, yrj, λj, sr, si, ≥ 0, ∀i, j, r; q free in sign,

where ε > 0 and s_r⁺ s_i^- and are slack variables. If we denote the optimal solution as (q₀^*,  λ₀^*, s₀^+*, s₀^-*), a DMU₀ is efficient if and only if the efficiency score q_o^*  = 1 and all s_io^{+* =} s_o^-* = 0. DMU₀ is weakly efficient if and only if q₀^*  = 1 but s_io^+* ≠ 0 or s_io^+* ≠ 0 for some i and r in some alternate optima (Cooper, Seiford and Zhu, 2011). Otherwise, a DMU is inefficient. Resulting from the optimal solution of the program (1) – (4), an inefficient DMU (x_o, y_o) can be projected to the BCC efficiency frontier as a combination of other DMU using the formulas:  and   (Cooper, Seiford and Zhu, 2011). Therefore, the lambdas allow us to identify the peer group of an inefficient DMU. By observing these efficient projections, we can analyze how a DMU should increase its outputs and/ or decrease its inputs to become relatively efficient.⁹

The period of analysis is divided into four subperiods: 2004-2006, 2007-2009, 2010-2012 and 2013-2015. The selection of the periods is mostly dictated by the availability of the data and the change in the data methodology. As explained in table 1, subperiods within 2004-2012 and subperiod 2013-2015 are characterized by different variables due to the availability of the data. Therefore, a direct comparison of results between periods is not advisable.

To circumvent the problem of missing data, we decided to calculate the simple three – years averages of data as the closest representative of the period. However, even this procedure resulted in some countries having missing data, so our approach was to exclude countries that had more than one missing data item. In order to keep as many countries as possible in the sample, those countries that had only one missing data item were kept in the sample and missing inputs/outputs were assigned a pessimistic value which is large/small enough for an objective function not to be entered, as proposed by Kuosmanen (2009). We did this only for countries that had one missing data item because we did not want to affect the “technology set” and worsen the relative ranking of other DMUs that had complete data. Additionally, we checked that the objective function in the solution included a multiplier of 0 for inputs/outputs variables with an arbitrary set value.

After the correction of the sample, the analysis includes 24 EU countries: Belgium, Bulgaria, the Czech Republic, Denmark, Germany, Estonia, Ireland, Spain, France, Croatia, Italy, Latvia, Lithuania, Hungary, the Netherlands, Austria, Poland, Portugal, Romania, Slovenia, Slovakia, Finland, Sweden, and the United Kingdom.

In the first step, we run the quantity-based models using variables expenditures (I) EX2 and financial aid (I)FA(%EX) as inputs and as outputs we use the percentage of graduates (O)GRAD(20-29), the education returns on labor market (O)U/UT and the percentage of highly educated population (O)POPT for the period of 2004-2012. We performed a similar analysis for the period 2013-2015, except that variable (I)FA(%EX) is replaced by the ratio of students per teacher (I)(S/T) and variable (O)GRAD(20-29) with (O)GRAD(20-34). As is obvious, such a selection of variables led to rewarding the quantity of the educational output and reporting on the efficiency of the tertiary education.

The second step was to include quality corrections for the previously obtained efficiency analysis. Firstly, we take account of output-quality and then we introduce the input-quality correction as well. For the output-quality control we replace the output variable (O)U/UT by the quality-corrected variable (O)U/UT*ACT ((O)U/UT multiplied with activity rates of the tertiary educated population). Also, variable (O)POPT was substituted for by (O)GDPpc in 2004-2012, and by (O)UR university ranking in 2013-2015 (as (O)GDPpc and (O)UR showed to be highly positively correlated). Afterward, the input-quality control was introduced by including PISA results in the analysis. Altogether we estimated 6 different models using inputs and outputs in certain subperiods as presented in table 2.

Table 2

Variables used in each DEA model, by period

DISPLAY Table

5 Results: analysis of the efficiency and effectiveness of tertiary education in the EU

6 Policy implications and future research recommendations

Appendix

Notes

2002

2006

2013a

2013b

2014

2009

2006

2011

* The authors would like to thank two anonymous referees for helpful comments on the paper. 

The article was submitted for the 2018 annual award of the Prof. Dr. Marijan Hanžeković Prize.

¹ Total enrolment in tertiary education (ISCED 5 to 8), regardless of age, expressed as a percentage of the total population of the five-year age group following on from secondary school leaving.

² We won't go in any details regarding the broader usage of DEA in public sector efficiency evaluations. However, interested reader can refer to the following research in this area: Clements (2002), Afonso and St. Aubyn (2006), Aristovnik (2013a; 2013b), Aristovnik and Obadić (2014) etc.

³ See Table A1 in Appendix.

⁴ For detailed explanation of variables see Aubyn et al. (2009).

⁵ Table A1 in Appendix provides a table with the previous research results.

⁶ See Table A1 in Appendix.

⁷ Due to data shortages Cyprus, Greece, Malta and Luxemburg were excluded from the dataset.

⁸ We excluded Cyprus, Malta, Luxemburg and Greece from the analysis due to the lack of data.

⁹ Some additional explanation on the BCC and other DEA models can be found in, for example, Cooper, Seiford and Tone (2006) or Cooper, Seiford and Zhu (2011).

Disclosure statement

References

1. Afonso, A. and St. Aubyn, M., 2006. Cross-country efficiency of secondary education provision: A semi-parametric analysis with non-discretionary inputs. Economic Modelling, 23(3), pp. 476–491 [CrossRef]


2. Ahec Šonje, A., Deskar-Škrbić, M. and Šonje, V., 2018. Efficiency of public expenditure on education: comparing Croatia with other NMS. MPRA Paper, No. 85152.


3. Altinok, N., Diebolt, C. and Demeulemeester, J.-L., 2014. A new international database on education quality: 1965–2010. Applied Economics, 46(11), pp. 1212–1247 [CrossRef]


4. Aristovnik, A., 2013a. Technical Efficiency of Education Sector in the EU and OECD Countries: The Case of Tertiary Education. Conference Proceedings 16^th Toulon-Verona Conference "Excellence in Services", Ljubljana, 29 – 30 August 2013, pp. 43–51.


5. Aristovnik, A., 2013b. Relative efficiency of education expenditures in Eastern Europe : a non-parametric approach. Journal of Knowledge Management, Economics and Information Technology, ScientificPapers.org, 3(3), pp. 1–4.


6. Aristovnik, A. and Obadić, A., 2011. The funding and efficiency of higher education in Croatia and Slovenia: a non-parametric comparison. Amfiteatru Economic, 13(30), pp. 362–376.


7. Aristovnik, A. and Obadić, A., 2014. Measuring relative efficiency of secondary education in selected EU and OECD countries: the case of Slovenia and Croatia. Technological and Economic Development of Economy, 20(3), pp. 419–433 [CrossRef]


8. Aubyn, M. S. [et al.], 2009. Study on the efficiency and effectiveness of public spending on tertiary education. European Economy, Economic Papers, No. 390.


9. Banker, R. D., Charnes, A. and Cooper, W. W., 1984. Some Models for Estimating Technical and Scale Inefficiencies in Data Envelopment Analysis. Management Science, 30(9), pp. 1078–1092.


10. Banker, R. D. and Morey, R. C., 1986. Efficiency Analysis for Exogenously Fixed Inputs and Outputs. Operations Research, 34(4), pp. 513–521 [CrossRef]


11. Barro, R., 2001. Human Capital and Growth. The American Economic Review, 91(2), pp. 12–17.


12. Barro, R., 2013. Education and Economic Growth. Annals of Economics and Finance, 14(2), pp. 301–328.


13. Bourguignon, F., Elkana, Y. and Pleskovic, B., 2007. Capacity Building in Economics Education and Research. Washington, DC: World Bank.


14. Charnes, A., Cooper, W. W. and Rhodes, E., 1978. Measuring the efficiency of decision making units. European Journal of Operational Research, 2(6), pp. 429–444 [CrossRef]


15. Clements, B. J., 2002. How efficient is education spending in Europe? European review of economics and finance, 1(1), pp. 3–26.


16. Cooper, W. W., Seiford, L. M. and Zhu, J. (eds.), 2011. Handbook on Data Envelopment Analysis. Cham: Springer Science and Business Media [CrossRef]


17. Cooper, W. W. and Seiford, L. M., Tone, K., 2006. Introduction to Data Envelopment Analysis and Its Uses. New York: Springer [CrossRef]


18. Emrouznejad, A. and Yang, G., 2018. A survey and analysis of the first 40 years of scholarly literature in DEA: 1978–2016. Socio-Economic Planning Sciences. Recent developments on the use of DEA in the public sector, 61(March), pp. 4–8 [CrossRef]


19. Eurostat, 2018a. Main GDP aggregates per capita. Gross domestic product at market prices. Luxembourg: Eurostat.


20. Eurostat, 2018b. Population by educational attainment level, sex and age (%) - main indicators. Luxembourg: Eurostat.


21. Eurostat, 2018c. Financial aid to students by education level - as % of total public expenditure. Luxembourg: Eurostat.


22. Eurostat, 2018d. General government expenditure by function (COFOG). Luxembourg: Eurostat.


23. Eurostat, 2018e. Ratio of pupils and students to teachers and academic staff by education level and programme orientation. Luxembourg: Eurostat.


24. Eurostat, 2018f. Tertiary education graduates. Luxembourg: Eurostat.


25. Eurostat, 2018g. Graduates by education level, programme orientation, completion, sex and age. Luxembourg: Eurostat.


26. Eurostat, 2018h. Unemployment rates by sex, age and educational attainment level (%). Luxembourg: Eurostat.


27. Eurostat, 2018i. Activity rates by sex, age and educational attainment level (%). Luxembourg: Eurostat.


28. Eurostat, 2018j. Underachieving 15-year-old students by sex and field - PISA survey. Luxembourg: Eurostat.


29. Fortunato, P. and Panizza, U., 2015. Democracy, education and the quality of government. Journal of Economic Growth, 20(4), pp. 333–363 [CrossRef]


30. Hanushek, E. A. and Kimko, D. D., 2000. Schooling, Labor-Force Quality, and the Growth of Nations. American Economic Review, 90(5), pp. 1184–1208 [CrossRef]


31. Kenny, J., 2008. Efficiency and Effectiveness in Higher Education: Who Is Accountable for What? Australian Universities’ Review, pp. 50(1), 11-19.


32. Kuosmanen, T., 2009. Data Envelopment Analysis with Missing Data. The Journal of the Operational Research Society, 60(12), pp. 1767–1774 [CrossRef]


33. Pritchett, L., 2001. Where Has All the Education Gone? The World Bank Economic Review, 15(3), pp. 367–391.


34. Pritchett, L., 2013. The Rebirth of Education: Schooling Ain’t Learning. Center for Global Development. Washington: Center for Global Development.


35. Scheerens, J., Luyten, H. and van Ravens, J., 2011. Measuring Educational Quality by Means of Indicators in: J. Scheerens, H. Luyten and J. van Ravens, eds. Perspectives on Educational Quality. Springer Netherlands: Dordrecht, pp. 35–50 [CrossRef]


36. Szirmai, A., 2015. Socio-Economic Development. Cambridge: Cambridge University Press.


37. Times Higher Education, 2017. World University Rankings.


38. Toth, R., 2009. Using DEA to Evaluate Efficiency of Higher Education. APSTRACT: Applied Studies in Agribusiness and Commerce, pp. 79–82.


39. World Bank, 2018. Education Statistics-Gross enrolment ratio in tertiary education. Washington: The World Bank.


40. Wößmann, L., 2006. Efficiency and Equity of European Education and Training Policies. CESifo Working Paper Series, (No. 1779).


41. Yotova, L. and Stefanova, K., 2017. Efficiency of Tertiary Education Expenditure in CEE Countries: Data Envelopment Analysis. Economic Alternatives, (3), pp. 352–364.

Afonso, A. and St. Aubyn, M., 2006. Cross-country efficiency of secondary education provision: A semi-parametric analysis with non-discretionary inputs. Economic Modelling, 23(3), pp. 476–491 [CrossRef]

Ahec Šonje, A., Deskar-Škrbić, M. and Šonje, V., 2018. Efficiency of public expenditure on education: comparing Croatia with other NMS. MPRA Paper, No. 85152.

Altinok, N., Diebolt, C. and Demeulemeester, J.-L., 2014. A new international database on education quality: 1965–2010. Applied Economics, 46(11), pp. 1212–1247 [CrossRef]

Aristovnik, A., 2013a. Technical Efficiency of Education Sector in the EU and OECD Countries: The Case of Tertiary Education. Conference Proceedings 16^th Toulon-Verona Conference "Excellence in Services", Ljubljana, 29 – 30 August 2013, pp. 43–51.

Aristovnik, A., 2013b. Relative efficiency of education expenditures in Eastern Europe : a non-parametric approach. Journal of Knowledge Management, Economics and Information Technology, ScientificPapers.org, 3(3), pp. 1–4.

Aristovnik, A. and Obadić, A., 2011. The funding and efficiency of higher education in Croatia and Slovenia: a non-parametric comparison. Amfiteatru Economic, 13(30), pp. 362–376.

Aristovnik, A. and Obadić, A., 2014. Measuring relative efficiency of secondary education in selected EU and OECD countries: the case of Slovenia and Croatia. Technological and Economic Development of Economy, 20(3), pp. 419–433 [CrossRef]

Aubyn, M. S. [et al.], 2009. Study on the efficiency and effectiveness of public spending on tertiary education. European Economy, Economic Papers, No. 390.

Banker, R. D., Charnes, A. and Cooper, W. W., 1984. Some Models for Estimating Technical and Scale Inefficiencies in Data Envelopment Analysis. Management Science, 30(9), pp. 1078–1092.

Banker, R. D. and Morey, R. C., 1986. Efficiency Analysis for Exogenously Fixed Inputs and Outputs. Operations Research, 34(4), pp. 513–521 [CrossRef]

Barro, R., 2001. Human Capital and Growth. The American Economic Review, 91(2), pp. 12–17.

Barro, R., 2013. Education and Economic Growth. Annals of Economics and Finance, 14(2), pp. 301–328.

Bourguignon, F., Elkana, Y. and Pleskovic, B., 2007. Capacity Building in Economics Education and Research. Washington, DC: World Bank.

Charnes, A., Cooper, W. W. and Rhodes, E., 1978. Measuring the efficiency of decision making units. European Journal of Operational Research, 2(6), pp. 429–444 [CrossRef]

Clements, B. J., 2002. How efficient is education spending in Europe? European review of economics and finance, 1(1), pp. 3–26.

Cooper, W. W., Seiford, L. M. and Zhu, J. (eds.), 2011. Handbook on Data Envelopment Analysis. Cham: Springer Science and Business Media [CrossRef]

Cooper, W. W. and Seiford, L. M., Tone, K., 2006. Introduction to Data Envelopment Analysis and Its Uses. New York: Springer [CrossRef]

Emrouznejad, A. and Yang, G., 2018. A survey and analysis of the first 40 years of scholarly literature in DEA: 1978–2016. Socio-Economic Planning Sciences. Recent developments on the use of DEA in the public sector, 61(March), pp. 4–8 [CrossRef]

Eurostat, 2018a. Main GDP aggregates per capita. Gross domestic product at market prices. Luxembourg: Eurostat.

Eurostat, 2018b. Population by educational attainment level, sex and age (%) - main indicators. Luxembourg: Eurostat.

Eurostat, 2018c. Financial aid to students by education level - as % of total public expenditure. Luxembourg: Eurostat.

Eurostat, 2018d. General government expenditure by function (COFOG). Luxembourg: Eurostat.

Eurostat, 2018e. Ratio of pupils and students to teachers and academic staff by education level and programme orientation. Luxembourg: Eurostat.

Eurostat, 2018f. Tertiary education graduates. Luxembourg: Eurostat.

Eurostat, 2018g. Graduates by education level, programme orientation, completion, sex and age. Luxembourg: Eurostat.

Eurostat, 2018h. Unemployment rates by sex, age and educational attainment level (%). Luxembourg: Eurostat.

Eurostat, 2018i. Activity rates by sex, age and educational attainment level (%). Luxembourg: Eurostat.

Eurostat, 2018j. Underachieving 15-year-old students by sex and field - PISA survey. Luxembourg: Eurostat.

Fortunato, P. and Panizza, U., 2015. Democracy, education and the quality of government. Journal of Economic Growth, 20(4), pp. 333–363 [CrossRef]

Hanushek, E. A. and Kimko, D. D., 2000. Schooling, Labor-Force Quality, and the Growth of Nations. American Economic Review, 90(5), pp. 1184–1208 [CrossRef]

Kenny, J., 2008. Efficiency and Effectiveness in Higher Education: Who Is Accountable for What? Australian Universities’ Review, pp. 50(1), 11-19.

Kuosmanen, T., 2009. Data Envelopment Analysis with Missing Data. The Journal of the Operational Research Society, 60(12), pp. 1767–1774 [CrossRef]

Pritchett, L., 2001. Where Has All the Education Gone? The World Bank Economic Review, 15(3), pp. 367–391.

Pritchett, L., 2013. The Rebirth of Education: Schooling Ain’t Learning. Center for Global Development. Washington: Center for Global Development.

Scheerens, J., Luyten, H. and van Ravens, J., 2011. Measuring Educational Quality by Means of Indicators in: J. Scheerens, H. Luyten and J. van Ravens, eds. Perspectives on Educational Quality. Springer Netherlands: Dordrecht, pp. 35–50 [CrossRef]

Szirmai, A., 2015. Socio-Economic Development. Cambridge: Cambridge University Press.

Times Higher Education, 2017. World University Rankings.

Toth, R., 2009. Using DEA to Evaluate Efficiency of Higher Education. APSTRACT: Applied Studies in Agribusiness and Commerce, pp. 79–82.

World Bank, 2018. Education Statistics-Gross enrolment ratio in tertiary education. Washington: The World Bank.

Wößmann, L., 2006. Efficiency and Equity of European Education and Training Policies. CESifo Working Paper Series, (No. 1779).

Yotova, L. and Stefanova, K., 2017. Efficiency of Tertiary Education Expenditure in CEE Countries: Data Envelopment Analysis. Economic Alternatives, (3), pp. 352–364.