A Review of Research and Development Evaluation Across the Globe

Nor Hasni Osman1*, Abu Bakar Allumi Nura1

  1. School of Technology Management and Logistics, College of Business, Universiti Utara Malaysia, Sintok, Kedah

Journal of Business Management and Accounting 5(1) (2015) · https://doi.org/10.32890/jbma2015.5.0.8862

Abstract

In view of the present rapid changing of technology and economic landscapes, the Research and Development (R&D) field has garnered more attention than before throughout the world. Industrial innovation in this knowledge-driven era demands R&D as the undisputable key for improvement and to maintain competitiveness. Parallel to the industrial and economic growths, more and more researchers and skilled workers are required to fill in the specific fields all over the globe. The growing numbers of R&D facilitiesand personnel, however, is unable to provide assurance that improvement would take place after a particular research result is implemented. R&D laboratories themselves are systems that process inputs and generate outputs, where in the process, inevitable challenges are encountered to ensure that the task is not only capable of solving particular problems, but also to ensure positive and innovative outcomes. R&D is also subject to performance analysis to measure the effectiveness, accomplishments, and productivity amongst others, because performance is an importance element in grant disbursement. In doing so, R&D evaluation is an important aspect and practised in most parts of the world with different methods and strategies. This paper reports an R&D overview and explores the need of the global community for post decision evaluation to ensure that the decision is sustainable in the long run.

Keywords: R&D, post decision evaluation, R&D lab, performance analysis

Introduction

Researchers have the task to utilise systematic and intensive studies to resolve specific problems from different angles and perspectives in order to embrace innovation, increase productivity, and profitability. Realising that knowledge has no boundaries and roams perpetually, many past and current theories that have been applied in research are easily busted, annulled, overruled, out-dated, and superseded in this challenging and competitive world. It is important to keep track of the ever changing scenarios, since it was revealed that even though a lot of R&D undertakings have been done, generally only a small amount or at some instances there is no research evaluation that follows afterwards. Most research studies conclude once they are implemented without due consideration given to the potentially infinite innovation that emerges every single day where knowledge is expanding and old ideas are often replaced with new ideas. Apart from China, Asian countries, particularly in Malaysia, Singapore, Japan and India, have recently seen an upsurge in R&D innovation where research has crossed borders, making them well known in the global R&D scene (Krishna, Patra, & Bhattacharya, 2012). The increasing numbers of R&D worldwide reveal that even though different methods and strategies are employed, there is still room for R&D improvement to enhance performance by measuring the effectiveness, efficiency, accomplishments, and productivity. As such R&D evaluation is an important aspect.

Research and development concept

R&D concept has been developed decades ago with various definition compilations subject to whatever backgrounds or purposes of the R&D. Generally, the character of R&D has been conceptualised as a systematic study. According to Godin (2001), National Science Board (2001), and UNESCO (1978), R&D is defined as an innovative and ingenious task that is carried out by employing systematic instruments to enrich piles of knowledge in devising latest applications. As for this study, R&D refers to any probe or work performed on a methodical and proficient approach to magnify the piles of knowledge in scientific, technological, managerial, and socio-political perspectives, with the objective of devising new concepts.

Post decision evaluation of R&D

A job might have been finished, but it has not necessarily been completed since finishing and completing are two different words with different meanings, purposes, and terms. R&D practices should be familiar with the term before, during, and after in the decision making process because measures should be taken before implementation, during implementation, and what happens after the implementation to avoid “fishing in muddy water”, as suggested by Kralj (2011). It was reported by Nill, Landwehr, Carabias, and Gérard (2009) that about 40% or more of the respondents declared not having their R&D evaluated and informed that the evaluation was in fact carried out before the implementation of the results. In some instances, most of the evaluation was performed internally. It is a rare phenomenon when they would evaluate the results after the implementation. If any, the evaluations mostly focused on the number of researchers, and the allocated funds and state revenues. Schainblatt (1982) indicated that in spite of billions of American dollars spent in R&D projects, only 20% of the R&D managers of major companies performed productivity appraisal of their R&D operations, and only a few number of them made assessment of any kind on profits generated from R&D investments. Certain engineers and scientists are cynical and afraid that their low and insufficient productivity will be revealed if the performance measure is done (Brown &

Svenson, 1998), leaving a delusional thought that all systems of performance measures are impractical, thus affiliating the practices of R&D as a “Lab”, which comprises input, process, and output elements. 4.0

The R&D lab as a system

Input, process, and output are elements that work in a system in an active organisation. Funds, manpower, data, concepts, instruments, amenities, and specific instructions are considered as the basic input. The R&D lab is the processing mechanism which converts the input into output; from preliminary draft proposals through performing research, hypotheses testing, until reaching the final revision. The new breakthrough whether it is a new concept, theory or idea, will later transpire as the output. The output ought to be measured and processed in terms of cost, quality, and quantity to assess its worthiness. Three foremost points in evaluating a decision as stated by Brumm (1986) are sustainability, holistic approach, and self-assurance stage. Sustainability is important since the finest concepts and proposals are destined to fail ultimately if the motivating forces are inconsistent as time goes by. It is only a premature success if no follow-up steps are taken. A holistic approach, on the other hand, is an art of expecting the unexpected by thinking outside the box. The last point is the self-assurance stage where the researchers must ensure that the R&D output resolves the related issues and confirms that the rendered decisions are either correct or reasonable decisions. Despite the fact that some of the ultimate obstacles encountered in measuring and evaluating R&D performance have been reviewed in some previous research, the most challenging task is to choose the best relevant set of correct measures that is coherent with the right subjects (Ojanen & Vuola, 2005). There are other problems encountered in measuring and evaluating R&D that need to be taken into consideration, such as contribution objective and expected period of completion. Brown and Svenson (1988) itemised four elements that instigate failure in R&D evaluation, i.e., extreme weight on internal measure, too much attention on behaviour, employing complicated measure procedure, and dearth of objective. Later, Kerssens-van Drongelen (1999) included acceptability problem into the list when he realised that experts argued the evaluations to be anti-productive while other argued that the measures are inappropriate or insufficient. Pappas and Remer (1985) added that this dissent is a setback to compete with.

R&D post decision reviews are often observed as an extra bureaucratic task where the willingness to cooperate with the application for revaluation work is usually at the bare minimum. If the required data are inconsistent with the data gathered at preceding checkpoints of the research, it will depreciate the project team’s trust in the company’s project management. As such, it is recommended that there should be a platform to ensure that effective post decision measurement systems exist to endorse R&D performance analysis so that the practice could be conducted at ease.

Rationale for R&D performance analysis

A measurement is only of value if utilised in a proper manner when making a decision. Lee, Son, and Lee (1996) claimed that evaluating R&D effectiveness is an undisputed factor in establishing whether such venture is tolerable or otherwise, and whether its highest output is accomplished and able to cope with the competition from new discoveries. Kerssens-van Drongelen and Cooke (1997) outlined two purposes of performance measurement; firstly, it motivates people, and secondly, to diagnose undertakings (e.g., projects) and units in an organisation. Furthermore, the studies by Kerssens-van Drongelen and Cooke (1997), Gold (1989), Meyer, Tertzakian, and Utterback (1997), and Loch and Tapper (2002) had classified R&D metrics according to the purpose of performance analysis in R&D.6.0 Dimensions of R&D performance analysis Although the number and the allocated amount of government grants are amongst the most common measure, what matters in the end is the accomplishment of the scientific objective. Berven (2011) insinuated that if the allocated amount of grants is treated to measure a performance, a decrease in the amount issued and the revaluation process might delay existing performance. It is necessary however to justify the actual allocated amount of grants and which specific field the R&D is designed. Chiesa, Coughlan, and Voss (1996) pointed out that when R&Ds begin to be responsible in their effectiveness and efficiency, as well as the likelihood for after decision assessment, it will uplift the awareness of both practitioners and academics (Brown & Svenson, 1988; Chiesa & Masella, 1996; Driva, Pawar, & Menon, 2000; Godener & Söderquist, 2004; Hauser, 1998; Ojanen & Vuola, 2005; Pappas & Remer, 1985; Pawar & Driva, 1999; Pillai & Rao, 1996; Poh, Ang, & Bai, 2001; Werner & Souder, 1997). Consequently, in view of the amplified attention in R&D performance measure, intellectuals began to explore the subject matter from different angles while clients focused more on planning in corporate and business strategies (Kumpe & Bolwijn, 1994; Pearson, Nixon, & Kerssens‐van Drongelen, 2000). Kim and Oh (2002) considered the performance measure ought to be dealt in two separate perspectives, i.e., who is the authorised person to conduct the task and how to deliver the task effectively. Cordero (1990) suggested the need to establish a practical affiliation between R&D performance and performance measure system that works in tandem. This would form the fundamentals that R&D productivity require as an efficient system to measure its performance.

Many researchers, found to concentrate on one analysis unit (Collier, DeMarco, & Fearey, 1996), were amongst the pioneers who assumed that performance could be measured by project-level metrics. Schainblatt (1981), Cooper and Kleinschmidt (1988), and Patterson (1983) were in agreement with this idea. Furthermore, Schainblatt (1981) and Wilson et al. (1994) recommended individual-level metrics, while Stahl and Steger (1977), and Moser (1985) recommended team-level metrics. Meanwhile, Cordero (1990) and Wacker (1998) suggested from-level metrics. In respect to “who is the authorised person to conduct the measurement”, Frost and Whitley (1971) established that it is the responsibility of the researcher himself or herself, together with his or her peers to perform the evaluation procedure, resolving that it is essential to engage several evaluation personnel instead of depending on the project directors and R&D heads. Stressing self-esteem as a factor, the process of measuring R&D performance is characteristically fragile (Keller & Holland, 1982). Wacker (1998) also agreed to engage several evaluators which include the researcher as well, external stakeholders, and his or her colleagues. With respect to “how to deliver the task effectively”, multiple citations are available that elaborate on the effect of more qualitative assessments on fostering an appropriate system in measuring performance. Barnowe (1975) recommended leadership as one of important criteria, while Wacker (1998) believed that subjective measures could be performed if the task is carried out by a group of evaluators comprising the researcher, his or her associates, and external auditors. Srinivas (2010) pointed out that when and how the evaluation takes place as deciding factors by indicating that the most preferable time to undertake the measure is when the research team remembers the most, i.e., soon after the delivery of the project and when all glitches have been taken care of. While still fresh, good supervision will generate new fruitful ideas. Nevertheless, it needs a stretched time frame along the process to justify the quality of the implementation before the result can materialise. This could be the adjusting period to make changes and to encounter challenges before the solution is finally found. Srinivas (2010) later explained further in “what to measure” factor by listing out the following terms: • • • • • •

Openness – honesty and transparent. Objectivity – stick to the main objective and avoid prejudice. Document success – proper document organisation. Look with hindsight – take heed on unknown risk. Being futuristic – concentrate on the prospect of the future. Reading between the lines – recognising pros and cons, and learn the positive and negative aspects.

The methods have been classified into quantitative-objective metrics, quantitativesubjective, and qualitative-subjective metrics (Wacker, 1998). These methods depend on the type of measure, either numerical or non-numerical, and whether it is based on objective information or subjective judgments. It was found that, integrated metric is the most fruitful method which combines multiple objective and subjective methods. Integrated metric, that consists of an interrelated but divisible set of qualitative and quantitative techniques, can be adaptably employed throughout all R&D categories. Along with Werner and Souder’s literature, there are some other research classifying R&D metrics by grouping them into the R&D type (Brown & Gobeli, 1992; Hauser & Zettelmeyer, 1997; Kim & Oh, 2002; Loch & Tapper, 2002; Werner & Souder, 1997). There are only about 20% of R&D research revaluated after termination, making it not of any importance (Zedtwitz, 2002). As a matter of fact, R&D organisations are generally convinced that there are vast advantages in post-decision revaluation, but apparently they are not making good use of the educational prospect. It was explained further that Menke (1997), who conducted a benchmark study of 79 premier R&D organisations, found less than 25% out of 79 organisations were fully engaged in post-project revaluations.

Table 1. Global R&D Evaluation Strategy
RegionCountryEvaluation StrategyWeakness of The Strategy
AsiaChina Singapore KoreaAssociates review panel Comprehensive variable evaluation Meta-analysis and in- depth reviewInstitutionalisation weakness and imbalance in review capacity The variables are associated to the environment and consistently differ Too rigid without contingency plans to accommodate incidental changes
EuropeUnited Kingdom TurkeyTarget recognition by government institutions Phrase anchored rating scaleToo bureaucratic and procedural Misleading, time consuming, and occasionally unreliable
SouthVenezuelaAnnual review usingDisconnection between the industry and the
Americanational innovation metricsuniversities in terms of R&D effort
AfricaEgypt South AfricaTask selection and generation Treatment effect analysisOptimisation is constrained and only solves non-complicated issues Too many fabricated positive rates

Too many fabricated positive rates

R&D evaluation status across the globe

R&D evaluations by all walks of professionals in different parts of the world have employed different methods and strategies. This paper provides a general outlook on how the performance evaluation is practised around the globe, which includes some Asian countries, two European nations, a South American nation, and two African countries. Table 1 illustrates the key points of evaluation strategy and the elaboration will follow afterwards. 6.1

Asia

China

A comparatively comprehensive framework is devised for R&D evaluation in China. Generally it is a desk study with the framework dimensions comprising objectives, goals, administration, implementation, effectiveness, and impact. This strategy is more qualitative than quantitative. The evaluation is reviewed by the associates of the peer review panel to determine the project approval. The review panel consists of a group of experts in numerous fields and disciplines. The weakness of this strategy is that although this approach is very meticulous and intricate; institutionalisation weakness causes ambiguity on which project should be reviewed and how; and low guidelines for specific implementation and imbalance in review capacity. These weaknesses indicate shortage of capable staff to man the evaluation which makes the review more concentrated upon the evaluation task itself than the results of the evaluation. Singapore

R&D evaluation in Singapore is performed mostly by embarking on a complete evaluation of a list of variables that is believed will potentially impact the selection and management of R&D. The variables are risk and return, strategic consideration, market exposure, resource and time based competitiveness constraint, and higher added value. The risk and return is an imperative variable since the real success and outcome of R&D remain unknown until they finally come into existence; when maximum return is the priority motive in decision making. As such, one of the best ways to gain competitive advantage is by reducing the allocated time from research to development. The next variable is strategic consideration which is founded on the strategic thinking skill because it has impact on funds and time obligation (Danila, 1989; Liberatore, 1987). The other stipulated variable is market exposure due to the fact that the prevailing upsurge productivity demands of high quality standard and ever increasing product innovation. The other variable refers to resources and time based competitiveness constraints, which refers to the scarcity of resources that will affect R&D decisions, progress, and etc. Higher added value is the other justified variable which refers to the engagement of unique technology in R&D innovation with added value, which will have a positive impact on the processing of products and the general development. In terms of weakness of the strategy, the main weakness is possibly the shortage of worldwide applicability because the variables are strongly attached to the environment, strategy, policy, and economic development that may not be suitable to other parts of the world (Liao & Greenfield, 2000). Korea Korea exercises R&D evaluation on two bases, i.e., the self-meta-analysis and in depth evaluation. The self-meta-analysis is a procedural strategy that evaluates the accomplishment of R&D corresponding to the performance scheme. The in depth analysis is used to identify problems confronted during implementation and concurrently rectify the problems to ensure effectiveness and efficiency. These strategies work in tandem. However, the weakness of this strategy is that it is too rigid and has no contingency plans where no account is taken on natural disaster, economic upturn and downturn, social upheaval, and political changes.

Europe

United Kingdom

In United Kingdom, the assessment of R&D is performed by setting the objectives that have to be achieved within the specified time to validate the fund amount taken. The government designs a speeding plan to be adhered to by all R&D organisations. The evaluation project is firstly selected by UK research council via traditional ex ante evaluations ahead of the handing of grants to safeguard a systematic review process. Upon approval, the R&D report is later submitted with matters ranging from the content of the research, income of staff members, publications, and etc. In spite of being very transparent and sequentially interlocked, this strategy is found to have too much procedure and characteristically too bureaucratic. Turkey Evaluating R&D in Turkey is conducted based on the classification of the research into three categories; curiosity driven (academic), customer driven (applied), and technology and innovation driven research. Each category is evaluated separately by five to eight reviewers who later will congregate as a panel to reach a final consensus. During evaluation, a phrase anchored rating scale is used. Even though the review process is simplified and reduces prejudice and variability of judgements, the reliability of the strategy is questionable. The apparent weakness is that it demands an exceptional observation talent and sufficient determination of analytical subjects. The strategy is also occasionally misleading and consumes too much time.

South America

Venezuela

In Venezuela, R&D evaluation is performed annually where the government draws up a list of national innovation metrics to assess the standard and competitiveness of R&D projects effectively. The evaluation is carried out in three different stages; innovation inputs, innovation outputs, and innovation outcomes. Innovation inputs are the deciding factor in the amount of funds and other support invested, while the innovation outputs observe whether the investment generates result and increases performance. Innovation outcomes focus on how many jobs and how much profits are created. The weakness of the strategy is that it does not perform as expected. The industry and university are not unified in the R&D effort whereas the assisted projects are not increasing as predicted. There is also lacking in funds for skills upgrade and training, and private companies are not interested in financing R&D projects.

Africa

Egypt

In Egypt, the main focus is project selection and generation to guarantee improved project development. The emphasis is to stimulate project selection models by using an effective communication system that is inter-connected to corporate priorities, implementation, strategy execution, and technical related issues. A project that is potentially viable technically and commercially will be projected and quantified by the prospective implementers and technologists, with the projecting and quantifying appraisals done from both the financial and marketing viewpoints. Schmidt and Freeland (1992) revealed that although this strategy was criticised to have constrained optimisation problem or as pointed out by Mitroff (1997) that it only solves simple R&D management matters, project selection strategy is one of the evaluation key elements and designs. South Africa As for R&D evaluation in South Africa, more consideration is given to treatment effect analysis based on harmonised data using non-parametric nearest-neighbour matching analysis. The weakness of this strategy is that the collected data depend on either sufficient or not the available nearest-neighbours, because it would result in fabricated statistics and false positive rates. 7.0

Concluding remarks

R&D needs a constant evaluation and post decision evaluation to measure its performance. Successfully implemented research tasks just end there and have never been kept track or re-evaluated, although there will always be space for improvement. This paper has elaborated on several evaluation strategies that have carried out around the world and explained some of these strategies and weaknesses. There are strategies that compare events before and after a change in policy is implemented, some strategies use parametric methods, and others employ non-parametric methods. Progressive R&D is actually functioning in a continuous circle and exposed in an ever-changing environment with constant innovation updates. The R&D “lab” itself is a system that processes inputs and generate outputs, where in the process inevitable challenges are encountered to ensure that the task not only solves particular problems, but also to ensure positive and innovative outcomes.

References

  1. Barnowe, J. T., (1975). Leadership and performance outcomes in research organisations: The supervisor of scientists as a source of assistance. Organisational Behavior and Human Performance, 14(2), 264-280.
  2. Berven, B., (2011). New DOE funding to accelerate ORNL advanced vehicle R&D. Energy and Environmental Sciences Quaterly, 3, 1-16.
  3. Brown, M. G., and Svenson, R. A., (1988). Measuring R&D productivity. Research technology management, 31(4), 11-15.
  4. Brown, M. G., and Svenson, R. A., (1998). Measuring R&D productivity. Research technology management, 41(6), 30.
  5. Brown, W. B., and Gobeli, D., (1992). Observations on the measurement of R&D productivity: a case study. Engineering Management, IEEE Transactions on, 39(4), 325-331.
  6. Brumm, J. H. J., (1986). Bureaucratic Competition and Weapon System Procurement. Defense Management Journal, 13-17.
  7. Chiesa, V., Coughlan, P., and Voss, C. A., (1996). Development of a technical innovation audit. Journal of Product Innovation Management, 13(2), 105-136.
  8. Chiesa, V., and Masella, C., (1996). Searching for an effective measure of R&D performance. Management Decision, 34(7), 49-57.
  9. Collier, B., DeMarco, T., and Fearey, P., (1996). A defined process for project post mortem review. Software, IEEE, 13(4), 65-72.
  10. Cooper, R. G., and Kleinschmidt, E. J., (1988). Resource allocation in the new product process. Industrial Marketing Management, 17(3), 249-262.
  11. Cordero, R., (1990). The measurement of innovation performance in the firm: An overview. Research Policy, 19(2), 185-192.
  12. Danila, N., (1989). Strategic evaluation and selection of R&D projects. R&D Management, 19(1), 47-62.
  13. Driva, H., Pawar, K., and Menon, U., (2000). Measuring product development performance in manufacturing organisations. International Journal of Production Economics, 63(2), 147-159.
  14. Frost, P. A., and Whitley, R., (1971). Communication patterns in research laboratory. R&D Management, 1(2), 71-79.
  15. Godener, A., and Söderquist, K. E., (2004). Use and impact of performance measurement results in R&D and NPD: an exploratory study. R&D Management, 34(2), 191-219.
  16. Godin, B., (2001). Tradition and Innovation: The Historical Contingency of R&D Statistical Classifications. Project on the History and Sociology of S&T Statistics, Working Paper(11).
  17. Gold, B., (1989). Some key problems in evaluating R&D performance. Journal of Engineering and Technology Management, 6(1), 59-70.
  18. Hauser, J. R., (1998). Research, development, and engineering metrics. Management Science, 44(12-part-1), 1670-1689.
  19. Hauser, J. R., and Zettelmeyer, F., (1997). Metrics to evaluate R, D&E. Research technology management, 40(4), 32.
  20. Keller, R. T., and Holland, W. E., (1982). The measurement of performance among research and development professional employees: A longitudinal analysis. Engineering Management, IEEE Transactions, 29(2), 54-58.
  21. Kerssens-van Drongelen, I. C., (1999). Systematic Design of R & D Performance Measurement Systems. (Doctoral Dissertation), University of Twente, Enschede, Netherlands.
  22. Kerssens-van Drongelen, I. C., and Cooke, A., (1997). Design principles for the development of measurement systems for research and development processes. R&D Management, 27(4), 345-357.
  23. Kim, B., and Oh, H., (2002). An effective R&D performance measurement system: survey of Korean R&D researchers. Omega, 30(1), 19-31.
  24. Kralj, B., (2011). Business decision making: the different way. http://www.bpmsecrets@ gmail.com
  25. Krishna, V., Patra, S. K., and Bhattacharya, S., (2012). Internationalisation of R&D and global nature of innovation: Emerging trends in India. Science Technology & Society, 17 (2), 165-199.
  26. Kumpe, T., and Bolwijn, P. T., (1994). Toward the innovative firm-Challenge for R&D management. Research technology management, 37(1), 38.
  27. Lee, M., Son, B., and Lee, H., (1996). Measuring R&D effectiveness in Korean companies. Research technology management, 39(6), 28.
  28. Liao, Z., and Greenfield, P. F., (2000). The synergy of corporate R&D and competitive strategies: An exploratory study in Australian high-technology companies. The Journal of High Technology Management Research, 11(1), 93-107.
  29. Liberatore, M. J., (1987). An extension of the analytic hierarchy process for industrial R&D project selection and resource allocation. Engineering Management, IEEE Transactions on(1), 12-18.
  30. Loch, C. H., and Tapper, U., (2002). Implementing a strategy‐driven performance measurement system for an applied research group. Journal of Product Innovation Management, 19(3), 185-198.
  31. Menke, M. M., (1997). Managing R&D for competitive advantage. Research technology management, 40(6), 40.
  32. Meyer, M.H., Tertzakian, P., and Utterback, J. M., (1997). Metrics for managing research and development in the context of the product family. Management Science, 43(1), 88-111.
  33. Mitroff, I., (1997). Uncommon Wisdom: Berrett-Koehler Publishers.
  34. Moser, M. R., (1985). Measuring performance in R&D settings. Research Management, 28(5), 31-33.
  35. National Science Board., (2001). Federal research resources: a process for setting priorities. Washington, DC.
  36. Nill, J., Landwehr, A., Carabias, V., and Gérard, C., (2009). Analysis of policy mixes to foster R&D investment and to contribute to the ERA. Luxembourg.
  37. Ojanen, V., and Vuola, O., (2005). Coping with the multiple dimensions of R&D performance analysis. International Journal of Technology Management, 33(2-3), 279-290.
  38. Pappas, R. A., and Remer, D. S., (1985). Measuring R&D productivity. Research-Technology Management, 28(3), 15-22.
  39. Patterson, W. C., (1983). Evaluating R&D performance at alcoa laboratories. Research Management, 26(2), 23-27.
  40. Pawar, K. S., and Driva, H., (1999). Performance measurement for product design and development in a manufacturing environment. International Journal of Production Economics, 60, 61-68.
  41. Pearson, A., Nixon, W., and Kerssens-van Drongelen, I. C., (2000). R&D as a business – what are the implications for performance measurement? R&D Management, 30(4), 355-366.
  42. Pillai, A. S., and Rao, K. S., (1996). Performance monitoring in R&D projects. R&D Management, 26(1), 57-65.
  43. Poh, K., Ang, B., and Bai, F., (2001). A comparative analysis of R&D project evaluation methods. R&D Management, 31(1), 63-75.
  44. Schainblatt, A. H., (1981). Measuring the productivity of scientists and engineers in R & D: a state of the practice review: The Urban Institute.
  45. Schainblatt, A. H., (1982). How companies measure the productivity of engineers and scientists. Research Management, 25(3), 10-18.
  46. Schmidt, R. L., and Freeland, J. R., (1992). Recent progress in modeling R&D project-selection processes. Engineering Management, IEEE Transactions on, 39(2), 189-201.
  47. Srinivas, H., (2010). Information for decision-making. http://www.mindtools.com
  48. Stahl, M. J., and Steger, J. A., (1977). Measuring innovation and productivity: a peer rating approach. Research Management, 20(1), 35-38.
  49. UNESCO., (1978, 28 November). Recommendation concerning the international standardization of statistics on science and techology. Paris.
  50. Wacker, J. G., (1998). A definition of theory: research guidelines for different theory-building research methods in operations management. Journal of Operations Management, 16(4), 361-385.
  51. Werner, B. M., and Souder, W. E., (1997). Measuring R&D performance--state of the art. Research technology management, 40(2), 34.
  52. Zedtwitz, M. V., (2002). The foreign R&D in China: The case of telecommunications. University of St Gallen-Rutgers. Tsinghua.