On the Use of the Novel Zorro Activation Functions in Long Short-Term Memory Neural Networks

Abstract

Activation functions are fundamental components of modern neural networks, including Large Language Models (LLMs). Nonlinear activations regulate the flow of information in most models and determine how data is processed. However, training or even fine-tuning very large and complex models with such activations, like those used in ChatGPT and DeepSeek, remains out of reach for many researchers. For this reason, traditional architectures like LSTM remain relevant for moderate or small-scale applications. In this context, the choice of appropriate functions can significantly influence a network's ability to learn complex patterns under limited computing resources. In this work, we analyze the behavior of a novel family of activation functions, called Zorro, which can serve as gating functions in LSTM architectures. We propose replacing traditional activation and gating functions in LSTMs with Zorro functions to improve model performance and convergence speed. Unlike conventional approaches, our method assigns a different function to each gate or activation, enabling the application of the same methodology to any gated architecture. We evaluate the modified LSTM models on widely used small-scale benchmark datasets, including Japanese Vowels and Human Activity Recognition for classification, as well as Chickenpox and Turbofan Degradation for regression. The results show that our method improves model accuracy by up to 10% and reduces training time by up to 15%.