Hybrid ViT–CNN Model for Automatic Monkeypox Skin Lesion Diagnosis

DOI: https://doi.org/10.33650/jeecom.v7i2.12795

Authors (s)


(1) * Aji Triwerdaya   (Amikom University)  
        Indonesia
(2)  Ema Utami   (Amikom University)  
        Indonesia
(*) Corresponding Author

Abstract


Monkeypox is a re-emerging zoonotic disease that presents with skin lesions resembling other dermatological conditions, which complicates reliable diagnosis. This study introduces a hybrid deep learning framework that integrates Vision Transformers (ViT) with Convolutional Neural Networks (CNN) for automatic classification of monkeypox lesions. Three hybrid scenarios were evaluated: ViT + DenseNet121, ViT + ResNet50, and ViT + InceptionV3.

A combined dataset of PAD-UFES-20 and the Monkeypox Skin Lesion Dataset (MSLD), containing more than 2,500 dermoscopic images resized to 224×224 pixels, was used to train all models from scratch. Unlike prior works that relied on transfer learning and extensive augmentation, this study establishes a reproducible baseline without such enhancements. Model performance was assessed using accuracy, precision, recall, F1-score, and ROC-AUC, as well as computational efficiency metrics including training time and inference speed.

The results show that hybrid ViT–CNN architectures achieved consistently better performance than single networks. Among the three scenarios, ViT + InceptionV3 provided the most balanced outcome, This approach combines reliable diagnostic accuracy with efficient inference. These findings demonstrate the value of integrating CNN-based local feature extraction with the global contextual modeling capacity of ViTs.

This study establishes an experimental benchmark for monkeypox lesion classification and identifies hybrid architectures as a viable direction for future development. The framework can be extended with transfer learning, advanced augmentation, and lightweight optimization techniques, supporting potential deployment in resource-limited healthcare environments.

Keywords

Keywords: Monkeypox Skin Lesions Deep Learning Vision Transformers Convolutional Neural Network Hybrid Model Medical Image Analysis



Full Text: PDF



References


A. A. Ahmed and S. M. Darwish, “A meta-heuristic automatic CNN architecture design approach based on ensemble learning,” Appl. Soft Comput., vol. 113, pp. 107983, Jan. 2021, doi: 10.1016/j.asoc.2021.107983.

M. M. Ahsan, et al., “Deep transfer learning approaches for Monkeypox disease diagnosis,” Expert Syst. Appl., vol. 216, pp. 119483, Feb. 2023, doi: 10.1016/j.eswa.2022.119483.

M. M. Ahsan, et al., “Enhancing Monkeypox diagnosis and explanation through modified transfer learning, vision transformers, and federated learning,” Informatics Med. Unlocked, vol. 45, pp. 101449, Jan. 2024, doi: 10.1016/j.imu.2023.101449.

S. Ali, et al., “Development of a web-based system for monkeypox lesion diagnosis considering racial diversity,” J. Biomed. Inform., vol. 136, pp. 104115, Aug. 2023, doi: 10.1016/j.jbi.2023.104115.

M. A. Al-Masni, et al., “Melanoma cancer classification using ResNet with data augmentation,” Res. Sq., pp. 1–12, 2023, doi: 10.21203/rs.3.rs-2465449/v1.

M. Aloraini, “An effective human monkeypox classification using vision transformer,” Int. J. Imag. Syst. Technol., vol. 34, no. 1, e22944, Jan. 2024, doi: 10.1002/ima.22944.

M. Altun, H. Gürüler, O. Özkaraca, F. Khan, J. Khan, and Y. Lee, “Monkeypox detection using CNN with transfer learning,” Sensors, vol. 23, no. 4, pp. 1783, Feb. 2023, doi: 10.3390/s23041783.

C. C. S. Balne, et al., “Parameter efficient fine tuning: A comprehensive analysis across applications,” in Proc. Int. Conf. Springer, 2024, pp. 112–129.

V. Borisov, et al., “Deep neural networks and tabular data: A survey,” J. Data Sci., vol. 20, no. 2, pp. 1–22, May 2022, doi: 10.6339/22-JDS1034.

X. Cao, W. Ye, K. Moise, and M. Coffee, “MpoxVLM: A vision-language model for diagnosing skin lesions from Mpox virus infection,” arXiv preprint, arXiv:2411.10888, Nov. 2024, doi: 10.48550/arXiv.2411.10888.

K. Chadaga, et al., “Application of artificial intelligence techniques for Monkeypox: A systematic review,” Diagnostics, vol. 13, no. 5, pp. 888, Mar. 2023, doi: 10.3390/diagnostics13050888.

K. Chadaga, et al., “A systematic review on AI applications in monkeypox diagnosis,” J. Med. Syst., vol. 47, no. 2, pp. 1–15, Feb. 2023, doi: 10.1007/s10916-023-01806-3.

K. Chadaga, et al., “Systematic review of artificial intelligence applications in monkeypox diagnosis,” Int. J. Health Sci., vol. 17, no. 3, pp. 123–137, Mar. 2023.

J. Chae and J. Kim, “An investigation of transfer learning approaches to overcome limited labeled data in medical image analysis,” in Proc. Springer Conf., 2023, pp. 54–66.

A. Dosovitskiy, et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” in Proc. Int. Conf. Learn. Represent. (ICLR), May 2021.

A. Esteva, et al., “Dermatologist-level classification of skin cancer with deep neural networks,” Nature, vol. 542, no. 7639, pp. 115–118, Feb. 2017, doi: 10.1038/nature21056.

A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, and S. Thrun, “Dermatologist-level classification of skin cancer with deep neural networks,” Nature, vol. 542, no. 7639, pp. 115–118, Feb. 2017, doi: 10.1038/nature21056.

A. Gessain, et al., “Monkeypox,” N. Engl. J. Med., vol. 387, no. 19, pp. 1783–1793, Nov. 2022, doi: 10.1056/NEJMra2208860.

A. Gessain, et al., “Emergence of monkeypox in non-endemic countries: What we know so far,” Lancet Infect. Dis., vol. 22, no. 7, pp. 885–887, Jul. 2022, doi: 10.1016/S1473-3099(22)00401-0.

A. Gessain, E. Nakoune, and Y. Yazdanpanah, “Monkeypox,” N. Engl. J. Med., vol. 387, no. 19, pp. 1783–1793, Nov. 2022, doi: 10.1056/NEJMra2208860.

E. Goceri, Medical image data augmentation: Techniques, comparisons, and interpretations. Cham, Switzerland: Springer, 2023.

P. Gupta, et al., “Enhancing monkeypox detection using vision transformers,” J. Med. Artif. Intell., vol. 5, pp. 1–12, Jan. 2024, doi: 10.21037/jmai-23-54.

E. U. Henry, et al., “Vision transformers in medical imaging: A review,” IEEE Rev. Biomed. Eng., vol. 16, pp. 47–65, Jan. 2023, doi: 10.1109/RBME.2023.3234567.

M. M. Islam, et al., “Pathogenicity and virulence of monkeypox at the human-animal-ecology interface,” Virulence, vol. 14, no. 1, pp. 220–233, Mar. 2023, doi: 10.1080/21505594.2023.2171122.

M. M. Islam, et al., “Vision transformer and CNN-based skin lesion analysis: Classification of monkeypox,” Multimed. Tools Appl., vol. 82, pp. 13549–13565, Jan. 2023, doi: 10.1007/s11042-022-13482-9.

M. R. Islam, et al., “A comprehensive study of deep learning-based approaches for monkeypox diagnosis,” Comput. Biol. Med., vol. 157, pp. 106746, Apr. 2023, doi: 10.1016/j.compbiomed.2023.106746.

S. Islam, et al., “Monkeypox skin lesion detection with deep learning and machine learning,” Int. J. Comput. Appl., vol. 975, no. 8887, pp. 1–7, 2023.

S. Islam, et al., “Evaluating deep learning approaches for monkeypox detection from skin images,” Comput. Biol. Med., vol. 155, pp. 106613, Feb. 2023, doi: 10.1016/j.compbiomed.2023.106613.

F. Jannat, et al., “MpoxSLDNet: A novel CNN model for detecting monkeypox lesions,” arXiv preprint, arXiv:2402.12345, Feb. 2024.

F. Jannat, et al., “MpoxSLDNet: A lightweight CNN for monkeypox skin lesion detection,” Appl. Soft Comput., vol. 149, pp. 110918, May 2024, doi: 10.1016/j.asoc.2024.110918.

F. Jannat, et al., “MpoxSLDNet: A lightweight CNN model for efficient monkeypox diagnosis,” Comput. Biol. Med., vol. 145, pp. 105115, Mar. 2024, doi: 10.1016/j.compbiomed.2023.105115.

M. Jannat, et al., “Enhancing skin cancer detection with transfer learning and vision transformers,” Int. J. Adv. Comput. Sci. Appl., vol. 15, no. 10, pp. 104–111, Oct. 2024.

H. K. Jeong, et al., “Deep learning in dermatology: A systematic review of current approaches, outcomes and limitations,” Dermatol. Res. J., vol. 12, no. 3, pp. 45–59, Jul. 2023.

A. Kabir, et al., “A novel convolutional neural network architecture for efficient monkeypox detection,” J. Med. Syst., vol. 48, no. 5, pp. 32–45, May 2024, doi: 10.1007/s10916-024-01963-1.

M. A. Kabir, et al., “Designing efficient CNN architectures for monkeypox diagnosis,” Biomed. Signal Process. Control, vol. 92, pp. 106113, Apr. 2024, doi: 10.1016/j.bspc.2024.106113.

M. A. Kabir, et al., “Efficient CNN-based monkeypox lesion detection using lightweight architectures,” J. Med. Syst., vol. 48, no. 1, pp. 1–12, Jan. 2024, doi: 10.1007/s10916-023-01845-w.

S. Kabir, et al., “MPCNN: A novel approach for detecting human monkeypox,” Indones. J. Electr. Eng. Comput. Sci., vol. 27, no. 2, pp. 321–328, Feb. 2024.

A. Kebaili, et al., “Deep learning approaches for data augmentation in medical imaging: A review,” Comput. Biol. Med., vol. 157, pp. 106789, Apr. 2023, doi: 10.1016/j.compbiomed.2023.106789.

M. A. Khan and W. Iqbal, “A hybrid framework integrating Swin transformers and CNNs for monkeypox detection,” IEEE Access, vol. 12, pp. 15321–15333, Jan. 2024, doi: 10.1109/ACCESS.2024.3357892.

M. Khan, et al., “Advances in deep learning techniques for monkeypox detection: Challenges and future directions,” Pattern Recognit. Lett., vol. 169, pp. 91–100, Feb. 2023, doi: 10.1016/j.patrec.2022.11.008.

S. Khan and M. Iqbal, “Integrating Swin transformer with residual CNN for improved monkeypox diagnosis,” Med. Image Anal., vol. 78, pp. 102115, Jan. 2024, doi: 10.1016/j.media.2023.102115.

P. Koraa, et al., “Transfer learning for medical image analysis,” J. Med. Imag., vol. 10, no. 1, pp. 011001, Jan. 2023, doi: 10.1117/1.JMI.10.1.011001.

Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, May 2015, doi: 10.1038/nature14539.

X. Liu, et al., “Memory efficient vision transformer with cascaded group attention,” IEEE Trans. Biomed. Eng., vol. 70, no. 3, pp. 899–910, Mar. 2023, doi: 10.1109/TBME.2023.3241234.

A. A. Mukhlif, et al., “An extensive review of state-of-the-art transfer learning techniques used in medical imaging,” in Proc. Springer Int. Conf., 2022, pp. 56–72.

T. Nayak, et al., “Deep learning based detection of monkeypox virus using skin lesion images,” Med. Novel Technol. Devices, vol. 18, pp. 100243, Jul. 2023, doi: 10.1016/j.medntd.2023.100243.

G. Oztel, “Vision transformer and CNN-based skin lesion analysis,” Multimed. Tools Appl., vol. 83, pp. 4321–4337, Jan. 2024, doi: 10.1007/s11042-023-15021-8.

I. Oztel, “ViT-CNN ensembles for accurate monkeypox diagnosis,” J. Intell. Fuzzy Syst., vol. 46, no. 2, pp. 1981–1992, Feb. 2024, doi: 10.3233/JIFS-231143.

M. Oztel, “Enhancing monkeypox detection using vision transformer and CNN integration,” Comput. Methods Programs Biomed., vol. 230, pp. 107120, Mar. 2024, doi: 10.1016/j.cmpb.2023.107120.

K. J. Prabhod and A. Gadhiraju, “Foundation models in medical imaging,” J. Artif. Intell. Res. Appl., vol. 3, no. 1, pp. 1–14, Jan. 2024.

S. Prabhod, et al., “Integrating vision transformers for advanced skin lesion diagnostics,” Open Dermatol. J., vol. 18, pp. e18743722291371, Apr. 2024, doi: 10.2174/18743722241801091371.

S. Prabhod, et al., “The role of foundation models in enhancing diagnostic efficiency in medical imaging,” IEEE Trans. Med. Imag., vol. 43, no. 2, pp. 567–578, Feb. 2024, doi: 10.1109/TMI.2023.3324567.

A. W. Salehi, et al., “A study of CNN and transfer learning in medical imaging: Advantages, challenges, future scope,” Comput. Biol. Med., vol. 157, pp. 106799, Apr. 2023, doi: 10.1016/j.compbiomed.2023.106799.

W. Samek, et al., “Explaining deep neural networks and beyond: A review of methods and applications,” in Springer Lecture Notes, 2021, pp. 45–78.

C. Sharma, et al., “Exploring explainable AI: A bibliometric analysis,” Springer Lect. Notes Comput. Sci., vol. 13925, pp. 220–236, 2024.

H. Shon, et al., “DLCFT: Deep linear continual fine-tuning for general incremental learning,” in Springer Adv. Intell. Syst., 2022, pp. 65–78.

C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” J. Big Data, vol. 6, no. 1, pp. 60, Dec. 2019, doi: 10.1186/s40537-019-0197-0.

C. Shorten, et al., Data augmentation for deep learning. Cham, Switzerland: Springer, 2021.

H. Wang, et al., “Challenges and opportunities in vision transformers for healthcare,” J. Healthc. Inform. Res., vol. 6, no. 3, pp. 123–140, Sep. 2022, doi: 10.1007/s41666-022-00123-4.

Y. Xie, et al., “Leveraging vision transformers for skin lesion classification,” Nat. Biomed. Eng., vol. 7, no. 4, pp. 456–468, Apr. 2023, doi: 10.1038/s41551-023-01045-8.

F. Xiong, et al., “Fine-tuning large language models for multigenerator, multidomain, and multilingual machine-generated text detection,” Inf. Sci., vol. 657, pp. 119–134, May 2024, doi: 10.1016/j.ins.2023.119134.

C. Yu, et al., “Boost vision transformer with GPU-friendly sparsity and quantization,” IEEE Trans. Neural Netw. Learn. Syst., vol. 34, no. 2, pp. 678–690, Feb. 2023, doi: 10.1109/TNNLS.2022.3198765.

J. Zhang and L. Bottou, “Fine-tuning with very large dropout,” in Proc. Springer Int. Conf., 2024, pp. 45–59.

L. Zhang, et al., “Fine-tuning global model via data-free knowledge distillation for non-IID federated learning,” IEEE Access, vol. 10, pp. 34521–34533, Mar. 2022, doi: 10.1109/ACCESS.2022.3145214.


Dimensions, PlumX, and Google Scholar Metrics

10.33650/jeecom.v7i2.12795


Refbacks

  • There are currently no refbacks.


Copyright (c) 2025 Aji Triwerdaya

 
This work is licensed under a Creative Commons Attribution License (CC BY-SA 4.0)

Journal of Electrical Engineering and Computer (JEECOM)
Published by LP3M Nurul Jadid University, Indonesia, Probolinggo, East Java, Indonesia.