Modelling stock prices using recurrent neural network and time series transformer

Abstract

Predicting stock market time series data is challenging due to its complexity, nonlinearity, and volatility. Recent advancements in deep learning algorithms have shown considerable promise for improving prediction accuracy by identifying intricate temporal patterns. This research employs three deep learning models—Long Short-Term Memory (LSTM), Gated Recurrent Units (GRU), and Temporal Fusion Transformers (TFT)—for stock price prediction. The models were assessed using stock price data from a high-volatility firms and optimized using Optuna and Nevergrad. Metrics such as Mean Error Square (MSE), Coefficient of determination (R2 score), Return on Investment (ROI) were used for evaluating the performance of the models. The comparative analysis revealed that the models optimized using Optuna surpassed those optimized with Nevergrad. The GRU Optuna-tuned model had a higher R2 score (94.81%), and projected a 1.5% ROI compared to the actual 3.5% in Vodacom Group Limited. In Bidvest Limited, the GRU Optuna-tuned had a higher R2 score of 96.07%, and projected a 50% ROI compared to the actual 52%. Meanwhile, the GRU Optuna-tuned model had R2 score (97.88%), and projected a -3.7% ROI compared to the actual ROI of -3.4% in Naspers Limited. In contrast, the TFT Nevergrad-tuned model exhibited an R2 score of 83%, and a projected ROI of -1.4%, compared to theactual ROI of 3.5% for Vodacom Group Limited. Similarly, in Bidvest Limited, the TFT Nevergrad-tuned model demonstrated a lower R2 score of 79%, and a projected ROI of 40%, relative to the actual ROI of 53%. The GRU Nevergrad tuned model showed an R2 score of -1151%, and a projected ROI of -25.84%, compared to the actual ROI of -3.4%. The Hyperbolic Tangent (Tanh) function in Vodacom Group Limited and Bidvest Limited assisted the GRU Optuna-tuned model in reducing MSE providing higher R2 score and ROI predictions that were close to the actual ROI. Meanwhile, the Parametric Leaky Tanh (PLTanh) in Naspers Limited lowered MSE and improved R2 score and ROI prediction that were increasingly aligned with real ROI. In contrast, the Sigmoid function exhibited higher MSE across all datasets for the three companies, indicating its inadequacy in managing non-linear data from the stock market when applied within the hidden layer. This work enhances existing research by providing findings on the effectiveness of different designs in financial forecasting tasks. This aids both practitioners and researchers in choosing suitable models for evolving market conditions.