A Machine Learning Classification Approach to Detect TLS-Based Malware Using Entropy-Based Flow Set Features
DOI:
https://doi.org/10.32890/jict2022.21.3.1Keywords:
Malware detection, machine learning, TLS, entropy, flow featuresAbstract
Transport Layer Security (TLS) based malware is one of the most hazardous malware types, as it relies on encryption to conceal connections. Due to the complexity of TLS traffic decryption, several anomaly-based detection studies have been conducted to detect TLS-based malware using different features and machine learning (ML) algorithms. However, most of these studies utilized flow features with no feature transformation or relied on inefficient flow feature transformations like frequency-based periodicity analysis and outliers percentage. This paper introduces TLSMalDetect, a TLS-based malware detection approach that integrates periodicity-independent entropy-based flow set (EFS) features generated by a flow feature transformation technique to solve flow feature utilization issues in related research. EFS features effectiveness was evaluated in two ways: (1) by comparing them to the corresponding outliers percentage and flow features using four feature importance methods, and (2) by analyzing classification performance with and without EFS features. Moreover, new Transmission Control Protocol features not explored in literature were incorporated into TLSMalDetect, and their contribution was assessed. This study’s results proved EFS features of the number of packets sent and received were superior to related outliers percentage and flow features and could remarkably increase the performance up to ~42% in the case of Support Vector Machine accuracy. Furthermore, using the basic features, TLSMalDetect achieved the highest accuracy of 93.69% by Naïve Bayes (NB) among the ML algorithms applied. Also, from a comparison view, TLSMalDetect’s Random Forest precision of 98.99% and NB recall of 92.91% exceeded the best relevant findings of previous studies. These comparative results demonstrated the TLSMalDetect’s ability to detect more malware flows out of total malicious flows than existing works. It could also generate more actual alerts from overall alerts than earlier research.Transport Layer Security (TLS) based malware is one of the most hazardous malware types, as it relies on encryption to conceal connections. Due to the complexity of TLS traffic decryption, several anomaly-based detection studies have been conducted to detect TLS-based malware using different features and machine learning (ML) algorithms. However, most of these studies utilized flow features with no feature transformation or relied on inefficient flow feature transformations like frequency-based periodicity analysis and outliers percentage. This paper introduces TLSMalDetect, a TLS-based malware detection approach that integrates periodicity-independent entropy-based flow set (EFS) features generated by a flow feature transformation technique to solve flow feature utilization issues in related research. EFS features effectiveness was evaluated in two ways: (1) by comparing them to the corresponding outliers percentage and flow features using four feature importance methods, and (2) by analyzing classification performance with and without EFS features. Moreover, new Transmission Control Protocol features not explored in literature were incorporated into TLSMalDetect, and their contribution was assessed. This study’s results proved EFS features of the number of packets sent and received were superior to related outliers percentage and flow features and could remarkably increase the performance up to ~42% in the case of Support Vector Machine accuracy. Furthermore, using the basic features, TLSMalDetect achieved the highest accuracy of 93.69% by Naïve Bayes (NB) among the ML algorithms applied. Also, from a comparison view, TLSMalDetect’s Random Forest precision of 98.99% and NB recall of 92.91% exceeded the best relevant findings of previous studies. These comparative results demonstrated the TLSMalDetect’s ability to detect more malware flows out of total malicious flows than existing works. It could also generate more actual alerts from overall alerts than earlier research.
References
Albright, D. (2021). Benchmarking average session duration: What it means and how to improve it. https://databox.com/average-session-duration-benchmark#definition
Anderson, B. H., & Mcgrew D. (2019). U.S. Patent No. 10,805,341: Leveraging point inferences on HTTP transactions for HTTPS malware detection. Google Patents. https://patents.google.com/ patent/US10805341B2/en
Anderson, B., & McGrew, D. (2016). Identifying encrypted malware traffic with contextual flow data. In Proceedings of the 2016 ACM Workshop on Artificial Intelligence and Security, Co-Located with CCS 2016 (pp. 35–46). https://doi.org/10.1145/2996758.2996768
Anderson, B., & McGrew, D. (2017). Machine learning for encrypted malware traffic classification: Accounting for noisy labels and non-stationarity. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Part F1296 (pp. 1723–1732). https://doi.org/10.1145/3097983.3098163
Anderson, B., Paul, S., & McGrew, D. (2018). Deciphering malware’s use of TLS (without decryption). Journal of Computer Virology and Hacking Techniques, 14(3), 195–211. https://doi.org/10.1007/s11416-017-0306-6
Bazuhair, W., & Lee, W. (2020). Detecting malign encrypted network traffic using perlin noise and convolutional neural network. In 10th Annual Computing and Communication Workshop and Conference, CCWC 2020 (pp. 200–206). https://doi.org/10.1109/CCWC47524.2020.9031116
Calderon, P., Hasegawa, H., Yamaguchi, Y., & Shimada, H. (2018). Malware detection based on HTTPS characteristic via machine learning. In Proceedings of the 4th International Conference on Information Systems Security and Privacy (ICISSP) (pp. 410–417). https://doi.org/10.5220/0006654604100417
Dai, R., Gao, C., Lang, B., Yang, L., Liu, H., & Chen, S. (2019). SSL malicious traffic detection based on multi-view features. ACM Journal of ICT, 21, No. 3 (July) 2022, pp: 279– International Conference Proceeding Series, 40–46. https://doi.org/10.1145/3371676.3371697
Fehrman, B., Woody, E., & Lillo, J. (2020). Detection of SSL/TLS malware beacons. Google Patents.
IDS 2017 | Canadian Institute for Cybersecurity | UNB. (n.d.). https://www.unb.ca/cic/datasets/ids-2017.html
Jenseg, O. (2019). A machine learning approach to detecting malware in TLS traffic using resilient network features (Master’s Thesis, NTNU). https://ntnuopen.ntnu.no/ntnu-xmlui/ handle/11250/2617735
Kato, H., Haruta, S., & Sasase, I. (2019). Android malware detection scheme based on level of SSL server certificate. In 2019 IEEE Global Communications Conference, GLOBECOM 2019 - Proceedings, 2 (pp. 379–389). https://doi.org/10.1109/ GLOBECOM38437.2019.9013483
Keshkeh, K., Jantan, A., Alieyan, K., & Gana, U. M. (2021). A review on TLS encryption malware detection: TLS features, machine learning usage, and future directions. Abdullah N., Manickam S., Anbar M. (Eds), Advances in Cyber Security. ACeS 2021. Communications in Computer and Information Science, 1487, 213–229. https://doi.org/10.1007/978-981-16-8059-5_13
Liu, J., Zeng, Y., Shi, J., Yang, Y., Wang, R., & He, L. (2019). Maldetect: A structure of encrypted malware traffic detection. Computers, Materials and Continua, 60(2), 721–739. https://doi.org/10.32604/cmc.2019.05610 Malware-Traffic-Analysis.net - Qbot (Qakbot) infection. (2020). https://www.malware-traffic-analysis.net/2020/01/29/index. html
Maroušek, J. (2017). Efficient kNN classification of malware from HTTPS data. https://dspace.cuni.cz/bitstream handle/ 20.500.11956/90345/BPTX_2016_1_11320_0_443594_0_184 381.pdf?sequence=1 MCFP Dataset (Malware Capture facility project - CTU University). (n.d.). https://mcfp.weebly.com/mcfp-dataset.html
Nagy, L. (2020). Nearly a quarter of malware now communicates using TLS – Sophos News. https://news.sophos.com/ en-us/2020/02/18/nearly-a-quarter-of-malware-now-communicates-using-tls/
Roques, O., Maffeis, S., & Cova, M. (2019). Detecting malware in TLS traffic (PhD dissertation, Imperial College London).
Rudolph, H. C., & Grundmann, N. (n.d.). Cipher Suite Info. https://ciphersuite.info/cs/ Journal of ICT, 21, No. 3 (July) 2022, pp: 279–
Seaman, C. (2019, July 2). Akamai Blog Anatomy of a SYN-ACK Attack. https://www.akamai.com/blog/security/anatomy-of-a-syn-ack-attack
Senecal, D., Kahn, A., Segal, O., … E. S.-U. P. A. 15, & 2019, U. (2019). Bot detection in an edge network using Transport Layer Security (TLS) fingerprint. Google Patents, 1.
Strasák, F. (2017). Detection of HTTPS malware traffic. Czech Technical University in Prague, Computing and Information Centre, May, 1–49.
Torroledo, I., Camacho, L. D., & Bahnsen, A. C. (2018). Hunting malicious TLS certificates with deep neural networks. In Proceedings of the ACM Conference on Computer and Communications Security (64–73). https://doi.org/10.1145/3270101.3270105
Tranalyzer - About. (n.d.). from https://tranalyzer.com/ Transport Layer Security (TLS) Parameters. (n.d.). https://www.iana. org/assignments/tls-parameters/tls-parameters.xhtml
Zheng, R., Liu, J., Liu, L., Liao, S., Li, K., Wei, J., Li, L., & Tian, Z. (2020). Two-layer detection framework with a high accuracy and efficiency for a malware family over the TLS protocol. PloS One, 15(5), e0232696. https://doi.org/10.1371/journal. pone.0232696
Published
Issue
Section
License
Copyright (c) 2022 Journal of Information and Communication Technology
This work is licensed under a Creative Commons Attribution 4.0 International License.
How to Cite
Research impact
Harvested 2026-09-05Counts differ between services because each indexes a different body of literature. None of them is the whole picture.
2002 - 2020
