Ensemble Meta Classifier with Sampling and Feature Selection for Data with Imbalance Multiclass Problem

Authors

  • Mohd Shamrie Sainin Faculty of Computing and Informatics, Universiti Malaysia Sabah, Malaysia
  • Rayner Alfred Faculty of Computing and Informatics, Universiti Malaysia Sabah, Malaysia
  • Faudziah Ahmad School of Computing, Universiti Utara Malaysia, Malaysia

DOI:

https://doi.org/10.32890/jict2021.20.2.6

Keywords:

Imbalance, multiclass, ensemble, feature selection, sampling

Abstract

Ensemble learning by combining several single or another ensemble classifier is one of the procedures to solve the imbalance problem in multiclass data. However, this approach is still facing the question of how the ensemble methods obtain their higher performance. In this paper, the investigation is carried out on the design of the ensemble meta classifier with sampling and feature selection for imbalance multiclass data. The specific objectives are 1) to improve the ensemble classifier through data-level approach (sampling and feature selection); 2) to perform experiments on sampling, feature selection, and ensemble classifier model; and 3) to evaluate the performance of the ensemble classifier. To fulfill the objectives, a preliminary data collection of Malaysian plants leaf images was prepared, experimented, and comparing the results. The ensemble design is also tested with another three high imbalance ratio benchmark data. It is found that the design using sampling, feature selection and ensemble classifier method using AdaboostM1 with Random Forest (also an ensemble classifier) provides the improved performance throughout the investigation. The result of this study is important to the ongoing problem of multiclass imbalance where specific structure and its performance can be improved in terms of processing time and accuracy.

References

Ali, H., Salleh, M. N. M., Saedudin, R., Hussain, K., & Mushtaq, M. F. (2019). Imbalance class problems in data mining: A review. Indonesian Journal of Electrical Engineering and Computer Science, 14(3), 1560–1571. https://doi.org/10.11591/ijeecs. v14.i3.pp1552-1563 Álvarez, J. D., Matias-Guiu, J. A., Cabrera-Martín, M. N., Risco-Martín, J. L., & Ayala, J. L. (2019). An application of machine learning with feature selection to improve diagnosis and classification of neurodegenerative disorders. BMC Bioinformatics, 20(491). https://doi.org/10.1186/s12859-019-3027-7

Barati, M., Abdullah, A., Mahmod, R., Mustapha, N., & Udzir, N. I. (2013). Features selection for IDS in encrypted traffic using genetic algorithm. In Proceedings of the 4th International Conference on Computing and Informatics (pp. 279–285). http://psasir.upm.edu.my/id/eprint/41307

Basir, M. A., Yusof, Y., & Hussin, M. S. (2018). Optimization of attribute selection model using bio-inspired algorithms. Journal of Information and Communication Technology, 18(1), 35–55.

Bia, J., & Zhang, C. (2017). An empirical comparison on state-of-the-art multi-class imbalance learning algorithms and a new diversified ensemble learning scheme. Knowledge-Based Systems, 158, 81–93. https://doi.org/10.1016/j.knosys.2018.05.037 Journal of ICT, 20, No. 2 (April) 2021, pp: 103–

Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2), 123–140. https://doi.org/10.1023/a:1018054314350

Cohen, I., Cozman, F. G., Sebe, N., Cirelo, M. C., & Huang, T. S. (2004). Semisupervised learning of classifiers: Theory, algorithms, and their application to human-computer interaction. IEEE Trans. Pattern Anal. Mach. Intell., 26, 1553–1567. https://doi.org/10.1109/TPAMI.2004.

Demisse, G. B., Tadesse, T., & Bayissa, Y. (2017). Data mining attribute selection approach for drought modelling: A case study for Greater Horn of Africa. International Journal of Data Mining & Knowledge Management Process, 7(4), 1–16. http://doi.org/10.5121/ijdkp.2017.7401

Dong, Q., Gong, S., & Zhu, X. (2019). Imbalanced deep learning by minority class incremental rectification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(6), 1367–1381. https://doi.org/10.1109/TPAMI.2018.2832629

Eschrich, S., Chawla, N. V., & Hall, L. O. (2002). Generalization methods in bioinformatics. In 2nd International Conference on Data Mining in Bioinformatics (BIOKDD’02) (pp. 25–32).

Feng, W., Huang, W., & Ren, J. (2018). Class imbalance ensemble learning based on the margin theory. Applied Science, 8(5), 815. https://doi.org/10.3390/app8050815

Freund, Y., & Schapire, R. (1996). Experiments with a new boosting algorithm. In International Conference on Machine Learning, Bari, Italy (pp. 148–156).

Galar, M., Fernandez, A., Barrenechea, E., Bustince, H., & Herrera, F. (2012). A review on ensembles for the class imbalance problem: Bagging-, boosting-, and hybrid-based approaches. IEEE Transactions on Systems, Man, and Cybernetics—Part C: Applications and Reviews, 42(4), 463–484. https://doi.org/10.1109/TSMCC.2011.2161285

Garcia, V., Sanchez, J. S., Mollineda, R. A., & Sotoca, J. M. (2007). The class imbalance problem in pattern classification and learning. In Tamida 2007, Saragossa, Spain (pp. 283–291).

Ghosh, S., Biswas, S., Sarkar, D., & Sarkar, P. P. (2014). A tutorial on different classification techniques for remotely sensed imagery datasets. Smart Computing Review, 4(1), 34–43. https://doi.org/10.6029/smartcr.2014.01.004

Gu, S., & Jin, Y. (2014). Generating diverse and accurate classifier ensembles using multi-objective optimization. IEEE Journal of ICT, 20, No. 2 (April) 2021, pp: 103– Symposium on Computational Intelligence in Multi-Criteria Decision-Making (MCDM), 9–15. https://doi.org/10.1109/ MCDM.2014.7007182

Guo, H., Diao, X., & Liu, H. (2019). Improving undersampling-based ensemble with rotation forest for imbalanced problem. Turkish Journal of Electrical Engineering & Computer Sciences, 27, 1371–1386. https://doi.org/10.3906/elk-1805-159

Hall, M. A. (1999). Correlation-based feature subset selection for machine learning (Unpublished Doctoral Thesis). The University of Waikato.

Hameed, S. S., Petinrin, O. O., Hashi, A. O., & Saeed, F. (2018). Filter-wrapper combination and embedded feature selection for gene expression data. Int. J. Advance Soft Compu. Appl, 10(1), 90–105.

Jegadeeshwaran, R., & Sugumaran, V. (2015). Health monitoring of a hydraulic brake system using nested dichotomy classifier – A machine learning approach. International Journal of Prognostics and Health Management, 6(1), 1–10.

Jerzy, B., Stefanowski, J., & Idkowiak, Ł. (2013). Extending bagging for imbalanced data. In Proceedings of the 8th International Conference on Computer Recognition Systems CORES (pp. 269–278). https://doi.org/10.1007/978-3-319-00969-8_26

Karthikeyan, T., & Thangaraju, P. (2013). Analysis of classification algorithms applied to hepatitis patients. International Journal of Computer Applications, 62(15), 25–30. https://doi.org/10.5120/10157-5032

Kohavi, R., & John, G. H. (1997). Wrappers for feature subset selection. Artificial Intelligence, 97(1–2), 273–324. https://doi.org/http://dx.doi.org/10.1016/S0004-3702(97)00043-X

Krawczyk, B. (2016). Learning from imbalanced data: Open challenges and future directions. Progress in Artificial Intelligence, 5(4), 221–232. https://doi.org/10.1007/s13748-016-0094-0

Kuncheva, L. I., & Rodrıguez, J. J. (2007). An experimental study on rotation forest ensembles. In Multiple Classifier Systems: 7th International Workshop, MCS 2007 (pp. 459–468). https://doi.org/10.1007/978-3-540-72523-7_46

Ladha, L., & Deepa, T. (2011). Feature selection methods and algorithms. International Journal on Computer Science and Engineering (IJCSE), 3, 1787–1797.

Langner, J. (2006). Neuronal network based recognition system of leaf images (Vol. 2009). http://www.jens-langner.de/lrecog/ Journal of ICT, 20, No. 2 (April) 2021, pp: 103–

Leathart, T., Pfahringer, B., & Frank, E. (2016). Building ensembles of adaptive nested dichotomies with random-pair selection. Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 179-194). https://doi.org/10.1007/978-3-319-46227-1_12

Leevy, J. L., Khoshgoftaar, T. M., Bauder, R. A., & Seliya, N. (2018). A survey on addressing high-class imbalance in big data. Journal of Big Data, 5(1), 42. https://doi.org/10.1186/s40537-018-0151-6

Li K., Zhou G., Zhai, J., Li, F., & Shao M. (2019). Improved PSO_ AdaBoost ensemble algorithm for imbalanced data. Sensor, 19(6), 1476. https://doi.org/10.3390/s19061476

Liu, H., & Setiono, R. (1996). A probabilistic approach to feature selection - A filter solution. In 13th International Conference on Machine Learning (pp. 319–327).

Mehra, N., & Gupta, S. (2013). Survey on multiclass classification methods. International Journal of Computer Science and Information Technologies, 4(4), 572–576.

Melville, P., & Mooney, R. J. (2004). Creating diversity in ensembles using artificial data. Information Fusion, 6, 99–111. https://doi.org/10.1016/j.inffus.2004.04.001

Mohsin, M. F. M., Hamdan, A. R., & Bakar, A. A. (2014). An evaluation of feature selection technique for dendrite cell algorithm. In Proceedings of the International Conference on IT Convergence and Security (ICITCS) (pp. 1–5). https://doi.org/10.1109/ICITCS.2014.7021732

Naghibi, S. A., Dolatkordestani, M., Rezaei, A., Amouzegari, P., Heravi, M. T., Kalantar, B., & Pradhan, B. (2019). Application of rotation forest with decision trees as base classifier and a novel ensemble model in spatial modeling of groundwater potential. Environmental Monitoring and Assessment, 191(248). https://doi.org/10.1007/s10661-019-7362-y

Nilashi, M., Ahmadi, H., Shahmoradi, L., Ibrahim, O., & Akbari, E. (2019). A predictive method for hepatitis disease diagnosis using ensembles of neuro-fuzzy technique. Journal of Infection and Public Health, 12(1), 13–20. https://doi.org/10.1016/j. jiph.2018.09.009

Onik, A. R., Haq, N. F., Alam, L., & Mamun, T. I. (2015). An analytical comparison on filter feature eextraction method in data mining using J48 classifier. International Journal of Computer Applications, 124(13), 1–8. https://doi.org/10.5120/ ijca2015905706 Journal of ICT, 20, No. 2 (April) 2021, pp: 103–

Rajagopal, S., Kundapur, P. P., & Hareesha, K. S. (2020). A stacking ensemble for network intrusion detection using heterogeneous datasets. Security and Communication Networks, 2020, 4586875. https://doi.org/10.1155/2020/4586875

Ren, Y., Zhang, L., & Suganthan, P. N. (2016). Ensemble classification and regression-recent developments, applications and future directions. IEEE Computational Intelligence Magazine, 11(1), 41–53. https://doi.org/10.1109/MCI.2015.2471235 Rodrı́guez, J. J., & Kuncheva, L. I. (2006). Rotation forest: A new classifier ensemble method. IEEE Transaction on Pattern Analysis and Machine Intelligence, 28(10), 1619–1621. https://doi.org/10.1109/TPAMI.2006.211

Samsuddin, S., Shah, Z. A., Saedudin, R. D. R., Kasim, S., & Seah, C. Sen. (2019). Analysis of attribute selection and classification algorithm applied to hepatitis patients. International Journal on Advanced Science, Engineering and Information Technology, 9(3), 967–971. http://dx.doi.org/10.18517/ijaseit.8.5.5041

Tasci, E. (2019). A meta-ensemble classifier approach: Random rotation forest. Balkan Journal of Electrical & Computer Engineering, 7(2), 182–187. https://doi.org/10.17694/bajece.502156

Triguero, I., del Río, S., López, V., Bacardit, J., Benítez, J. M., & Herrera, F. (2015). ROSEFW-RF: The winner algorithm for the ECBDL’14 big data competition: An extremely imbalanced big data bioinformatics problem. Knowledge-Based Systems, 87, 69–79. https://doi.org/https://doi.org/10.1016/j. knosys.2015.05.027

Wang, S., & Yao, X. (2012). Multiclass imbalance problems: Analysis and potential solutions. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 2(4), 1119–1130. https://doi.org/10.1109/TSMCB.2012.2187280

Webb, G. I. (2000). MultiBoosting: A technique for combining boosting and wagging. Machine Learning, 40, 159–196. https://doi.org/10.1023/A:1007659514849

Wever, M., Mohr, F., & Hüllermeier, E. (2018). Ensembles of evolved nested dichotomies for classification. In Proceedings of the Genetic and Evolutionary Computation Conference, 561–568. https://doi.org/10.1145/3205455.3205562

Wolpert, D. H. (1992). Stacked generalization. Neural Networks, 5(2), 241–260. https://doi.org/10.1016/S0893-6080(05)80023-1

Downloads

Published

21-02-2021

Issue

Vol. 20 No. 2 (2021): Journal of Information and Communication Technology (JICT) Vol.20, No.2, April 2021

Section

Articles

How to Cite

Sainin, M. S., Alfred, R., & Ahmad, F. (2021). Ensemble Meta Classifier with Sampling and Feature Selection for Data with Imbalance Multiclass Problem. Journal of Information and Communication Technology, 20(2), 103-133. https://doi.org/10.32890/jict2021.20.2.6

Research impact

Harvested 2026-09-06
0 citations recorded so far

Counts differ between services because each indexes a different body of literature. None of them is the whole picture.

Identifiers DOI 10.32890/jict2021.20.2.6

Most read articles by the same author(s)