A comparison of small sample methods for Handshape Recognition

Franco Ronchetti; Facundo Quiroga; Ulises Jeremias Cornejo Fandos; Gastón Gustavo Rios; Pedro Dal Bianco; Waldo Hasperué; Laura Lanzarini

doi:10.24215/16666038.23.e03

Authors

Franco Ronchetti Instituto de Investigación en Informática LIDI, Universidad Nacional de La Plata, Facultad de Informática, UNLP – CIC https://orcid.org/0000-0003-3173-1327
Facundo Quiroga Instituto de Investigación en Informática LIDI, Universidad Nacional de La Plata, Facultad de Informática, UNLP – CIC https://orcid.org/0000-0003-4495-4327
Ulises Jeremias Cornejo Fandos Instituto de Investigación en Informática LIDI, Universidad Nacional de La Plata, Facultad de Informática, UNLP – CIC https://orcid.org/0000-0003-1474-2083
Gastón Gustavo Rios Instituto de Investigación en Informática LIDI, Universidad Nacional de La Plata, Facultad de Informática, UNLP – CIC https://orcid.org/0000-0003-0252-7036
Pedro Dal Bianco Instituto de Investigación en Informática LIDI, Universidad Nacional de La Plata, Facultad de Informática, UNLP – CIC https://orcid.org/0000-0001-7197-8602
Waldo Hasperué Instituto de Investigación en Informática LIDI, Universidad Nacional de La Plata, Facultad de Informática, UNLP – CIC https://orcid.org/0000-0002-9950-1563
Laura Lanzarini Instituto de Investigación en Informática LIDI, Universidad Nacional de La Plata, Facultad de Informática, UNLP – CIC https://orcid.org/0000-0001-7027-7564

DOI:

https://doi.org/10.24215/16666038.23.e03

Keywords:

sign language, handshape recognition, DenseNet, prototypical networks, MAML, transfer learning, small datasets

Abstract

Automatic Sign Language Translation (SLT) systems can be a great asset to improve the communication with and within deaf communities. Currently, the main issue preventing effective translation models lays in the low availability of labelled data, which hinders the use of modern deep learning models.

SLT is a complex problem that involves many subtasks, of which handshape recognition is the most important. We compare a series of models specially tailored for small datasets to improve their performance on handshape recognition tasks. We evaluate Wide-DenseNet and few-shot Prototypical Network models with and without transfer learning, and also using Model-Agnostic Meta-Learning (MAML).

Our findings indicate that Wide-DenseNet without transfer learning and Prototipical Networks with transfer learning provide the best results. Prototypical networks, particularly, are vastly superior when using less than 30 samples, while Wide-DenseNet achieves the best results with more samples. On the other hand, MAML does not improve performance in any scenario. These results can help to design better SLT models.

Downloads

Download data is not yet available.

References

O. Koller, “Quantitative survey of the state of the art in sign language recognition,” CoRR, vol. abs/2008.09918, 2020.

D. Bragg, O. Koller, M. Bellard, L. Berke, P. Boudreault, A. Braffort, N. Caselli, M. Huenerfauth, H. Kacorri, T. Verhoef, et al., “Sign language recognition, generation, and translation: An interdisciplinary perspective,” in The 21st International ACM SIGACCESS Conference on Computers and Accessibility, pp. 16–31, 2019.

A. A. I. Sidig, H. Luqman, and S. A. Mahmoud, “Arabic sign language recognition using vision and hand tracking features with hmm,” International Journal of Intelligent Systems Technologies and Applications, vol. 18, no. 5, pp. 430–447, 2019.

W. Min, W. Ya, and Z. Xiao-Juan, “An improved adaptation algorithm for signer-independent sign language recognition,” International Journal of Intelligent Systems Technologies and Applications, vol. 17, no. 4, pp. 427–438, 2018.

O. Koller, H. Ney, and R. Bowden, “Deep hand: How to train a cnn on 1 million hand images when your data is continuous and weakly labelled,” in IEEE Conference on Computer Vision and Pattern Recognition, (Las Vegas, NV, USA), pp. 3793–3802, June 2016.

J. Snell, K. Swersky, and R. S. Zemel, “Prototypical networks for few-shot learning,” CoRR, vol. abs/1703.05175, 2017.

G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.

C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” CoRR, vol. abs/1703.03400, 2017.

O. Koller, H. Ney, and R. Bowden, “Deep hand: How to train a cnn on 1 million hand images when your data is continuous and weakly labelled,” in IEEE Conference on Computer Vision and Pattern Recognition, (Las Vegas, NV, USA), pp. 3793–3802, June 2016.

F. Ronchetti, F. Quiroga, L. Lanzarini, and C. Estrebou, “Handshape recognition for argentinian sign language using probsom,” Journal of Computer Science and Technology, vol. 16, no. 1, pp. 1–5, 2016.

F. Quiroga, R. Antonio, F. Ronchetti, L. C. Lanzarini, and A. Rosete, “A study of convolutional architectures for handshape recognition applied to sign language,” in XXIII Congreso Argentino de Ciencias de la Computaci´on (La Plata, 2017)., 2017.

D. N´u˜nez Fern´andez and B. Kwolek, “Hand posture recognition using convolutional neural network,” in Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications (M. Mendoza and S. Velast ´ın, eds.), (Cham), pp. 441–449, Springer International Publishing, 2018.

A. A. Alani, G. Cosma, A. Taherkhani, and T. M. McGinnity, “Hand gesture recognition using an adapted convolutional neural network with data augmentation,” 2018 4th International Conference on Information Management (ICIM), pp. 5–12, 2018.

A. Tang, K. Lu, Y. Wang, J. Huang, and H. Li, “A real-time hand posture recognition system using deep neural networks,” ACM Transactions on Intelligent Systems and Technology (TIST), vol. 6, no. 2, p. 21, 2015.

P. Barros, S. Magg, C. Weber, and S. Wermter, “A multichannel convolutional neural network for hand posture recognition,” in International Conference on Artificial Neural Networks, pp. 403–410, 09 2014.

S. Ameen and S. Vadera, “A convolutional neural network to classify american sign language fingerspelling from depth and colour images,” Expert Systems, vol. 34, p. e12197, February 2017.

U. J. Cornejo Fandos, G. G. Rios, F. Ronchetti, F. Quiroga, W. Hasperu´e, and L. C. Lanzarini, “Recognizing handshapes using small datasets,” in XXV Congreso Argentino de Ciencias de la Computaci´on (CACIC 2019, Universidad Nacional de R´ıo Cuarto), 2019.

C. Tan, F. Sun, T. Kong, W. Zhang, C. Yang, and C. Liu, “A survey on deep transfer learning,” in Artificial Neural Networks and Machine Learning – ICANN 2018 (V. K˚urkov´a, Y. Manolopoulos, B. Hammer, L. Iliadis, and I. Maglogiannis, eds.), (Cham), pp. 270–279, Springer International Publishing, 2018.

H. Pham, M. Guan, B. Zoph, Q. Le, and J. Dean, “Efficient neural architecture search via parameters sharing,” in Proceedings of the 35th International Conference on Machine Learning (J. Dy and A. Krause, eds.), vol. 80 of Proceedings of Machine Learning Research, (Stockholmsm assan, Stockholm Sweden), pp. 4095–4104, PMLR, 10–15 Jul 2018.

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 2015.

J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7132–7141, 2017.

A. Farhadi, D. Forsyth, and R. White, “Transfer learning in sign language,” in 2007 IEEE Conference on Computer Vision and Pattern Recognition, 06 2007.

U. Cˆot´e-Allard, C. L. Fall, A. Campeau-Lecours, C. Gosselin, F. Laviolette, and B. Gosselin, “Transfer learning for semg hand gestures recognition using convolutional neural networks,” in 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 1663–1668, 2017.

K. Weiss, T. Khoshgoftaar, and D. Wang, “A survey of transfer learning,” Journal of Big Data, vol. 3, 12 2016.

A. Krizhevsky, G. Hinton, et al., “Learning multiple layers of features from tiny images,” tech. rep., CIFAR, 2009.

Y. LeCun and C. Cortes, “MNIST handwritten digit database,” tech. rep., MNIST, 2010.

D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 2014. cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.