Yield and Quality Prediction of Crops using a Deep Learning-Based Multimodal Data Integration Framework

A precise prediction of crop yield and quality is essential for improving resource efficiency, optimising supply chain management, and enhancing food security, particularly in the face of climate change and resource constraints. This paper proposes a deep learning-based multimodal integration framework that combines Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM) networks, and an attention mechanism to predict yield and quality jointly. The framework integrates heterogeneous data sources, including soil properties, irrigation records, microclimate variables, and vegetation indices (NDVI/EVI) derived from remote sensing imagery. Simulation experiments were conducted on rice and tomato datasets under both normal and stressful conditions, including six representative scenarios: drought stress, excessive irrigation, heat waves, fertiliser mismanagement, pest and disease outbreaks, and soil salinity. Results indicate that the proposed model consistently outperforms baseline approaches, achieving an RMSE of 245 kg/ha and an R2 of 0.93 for rice yield, and an MAE of 0.15% and an R2 of 0.92 for tomato quality. The attention mechanism further enhances interpretability by identifying critical growth stages and influential features. These findings confirm the robustness and practical relevance of the proposed framework for precision and climate-smart agriculture, providing a comprehensive tool to optimise productivity and post-harvest quality simultaneously.

Introduction

Accurate prediction of crop yield and quality is essential for enhancing resource efficiency, optimizing supply chain management, and ensuring global food security, especially as challenges like climate change, soil degradation, and water scarcity accelerate. Traditional approaches, including statistical regression models and process-based crop growth simulators such as DSSAT and APSIM, have been widely used over the past decades to estimate crop performance [1]–[3]. Although these models provide valuable insights, they often require extensive calibration and site-specific field data, and they tend to underperform in capturing nonlinear and complex interactions among climatic, soil, and management variables [4]–[6].

Recent advances in artificial intelligence (AI) and deep learning (DL) have opened new opportunities for improving agricultural prediction and decision-making. Convolutional Neural Networks (CNNs) have demonstrated strong capability in extracting spatial and spectral features from multispectral or UAV imagery. In contrast, Long Short-Term Memory (LSTM) networks excel at learning temporal dependencies from meteorological and IoT sensor data. More recently, Transformer-based and attention-driven architectures have further enhanced the ability to focus selectively on critical features within complex multimodal datasets [7]–[9]. A growing body of literature has applied deep learning to agricultural yield forecasting. For example, an increasing body of literature has applied deep learning. Based on the findings of previous studies [15], [16], CNN–LSTM hybrid architectures have been successfully used to model spatio-temporal interactions between vegetation indices and weather variables. Other studies integrated soil and climate data to improve generalization under heterogeneous growing environments [10]–[12].

Despite such progress, several gaps remain. First, most existing studies have focused primarily on yield prediction, while crop quality—an equally critical factor for market value and nutritional assessment—has received less attention [13], [14]. Second, most models rely on single-source data, such as satellite imagery or meteorological records, limiting their adaptability across different regions and cropping systems [15], [16]. Third, the comparative effectiveness of multimodal fusion strategies (early, late, and hybrid) remains insufficiently explored, and empirical evaluation under realistic field stress conditions is lacking.

To address these limitations, this study proposes a deep learning–based multimodal data integration framework that combines CNN, LSTM, and an attention mechanism to predict both crop yield and quality jointly. The framework integrates heterogeneous datasets—soil properties, irrigation history, microclimate variables, and remote sensing indices (NDVI/EVI)—collected from rice and tomato farms in Vietnam. Unlike previous yield-only approaches, the proposed model aims to capture the synergistic effects of soil–plant–climate interactions, while enhancing interpretability through the attention mechanism.

The contributions of this study are threefold:

-A novel multimodal deep learning architecture has been developed to simultaneously predict crop yield and quality by integrating soil, irrigation, climate, and remote sensing data.

-The model introduces an attention-based fusion strategy that dynamically identifies the most influential variables and growth stages, improving both accuracy and interpretability.

-Extensive simulation experiments on rice and tomato datasets—including six representative stress scenarios (drought, over-irrigation, heatwave, fertilizer mismanagement, pest/disease outbreak, and salinity)—demonstrate the robustness and generalizability of the proposed framework under real-world conditions.

The remainder of this paper is organized as follows. Section 2 describes the framework design and multimodal data integration strategy. Section 3 outlines the dataset, preprocessing, and simulation setup. Section 4 presents and discusses the results, including robustness analysis and model interpretability. Section 5 concludes with key findings and directions for future research.

Method

Data Collection

To ensure accurate prediction of both crop yield and quality, it is vital to integrate diverse datasets that capture soil, climate, management, and plant growth conditions. Therefore, this study combines data from multiple sources, including field sensors, farm records, meteorological services, and remote sensing platforms, to develop a comprehensive dataset for model building. In this study, data were collected from various sources, such as field sensors, farmer records, meteorological services, and open-access databases. Soil parameters like pH, EC, and soil moisture were measured using IoT sensors or extracted from ISRIC Soil Grids [1]. Irrigation schedules and water volumes were documented through farmer logs or flow sensors. Microclimate variables, including temperature, humidity, and solar radiation, were collected from VNMHA stations and supplemented with data from NASA POWER and OpenWeatherMap APIs [2], [3]. Remote sensing data, including NDVI and EVI indices, were derived from Sentinel-2 (Copernicus) and MODIS (NASA) imagery processed on Google Earth Engine [4]–[6]. Crop management information, including varieties, fertilizer applications, and cultivation practices, was obtained from farmer records and projects conducted by VAAS and IRRI in Vietnam’s Mekong Delta. Table I provides a summary of the variables, sources, units, frequency, and study locations.

Table I. Summary of the Variables, Sources, Units, Frequency, and Study Locations

Data type	Variables	Source	Unit	Frequency	Location
Soil properties	pH, EC, Soil moisture	IoT sensors, Soil Grids (ISRIC)	pH, dS/m, %	30 min – 1h	Dong Thap, An Giang (rice); Lam Dong, Hanoi (tomato)
Irrigation	Irrigation schedule, Water volume	Farmer logs, Flow sensors, FAO CROPWAT	Liters/hour	Per irrigation event	Same as above
Microclimate	Air temperature, Humidity, Solar radiation	VNMHA stations, NASA POWER, Open Weather Map	°C, %, W/m2	10 min – 1h	All study areas
Remote sensing	NDVI, EVI (Sentinel-2, MODIS)	ESA Copernicus Hub, NASA MODIS, Google Earth Engine	Index (–1 to 1)	5 days (Sentinel-2), 16 days (MODIS)	All study areas
Management/Genetic Info	Crop variety, Fertilizer, Cultivation practices	Farmer records, VAAS, IRRI projects	–	Per season	Mekong Delta (rice), Lam Dong & Hanoi (tomato)

Data Processing

Before using the data in the deep learning framework, all datasets were pre-processed to ensure consistency and reliability. First, temporal and spatial synchronization was achieved by interpolating, resampling, or georeferencing sensor, meteorological, and remote sensing data to a standard timeline and study area. Input variables were then normalized using Min–Max scaling or Z-score standardization, with missing values imputed and cloud-affected Sentinel-2 images filtered through cloud masking. Finally, feature extraction and transformation were applied: NDVI and EVI indices were calculated from satellite imagery, PCA or autoencoders were used for dimensionality reduction, lag features and Fourier transforms were generated from climatic time series, and soil and irrigation data were aggregated into daily summaries or sequential inputs for the LSTM. These steps produced harmonized and robust data optimized for model training.

To standard heterogeneous sources, various preprocessing techniques were employed, including resampling, normalization, missing-value imputation, and noise removal, as summarized in Table II. These steps produced harmonised datasets optimised for model training and evaluation.

Table II. Preprocessing Techniques Applied to Each Data Type

Data type	Techniques applied	Purpose
Soil & Irrigation	Resampling, Min–Max scaling, KNN-imputation	Synchronize, normalize, handle missing
Microclimate	Z-score standardization, lag features, rolling mean	Reduce bias, highlight temporal trends
Remote sensing	Cloud masking (Fmask), NDVI/EVI calculation, PCA	Remove noise, extract vegetation info
Time Series (all)	Interpolation, Fourier transform, autoencoder	Fill gaps, capture periodic patterns

Deep Learning Model

A hybrid deep learning model integrates spatial, temporal, and contextual information to forecast crop yield and quality. It employs Convolutional Neural Networks (CNNs) for spatial features, Long Short-Term Memory (LSTM) networks for temporal dependencies, and an attention mechanism for feature weighting.

Fig. 1 shows the training process: multimodal inputs (soil, irrigation, climate, and remote sensing indices) are pre-processed and then passed through a CNN, LSTM, and attention layer. The combined representation enables dual predictions of yield (kg/ha) and quality (0–1 or %), optimized with a joint loss function.

Fig. 1. Training flowchart of the proposed CNN–LSTM–Attention model.

The proposed framework combines Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM) networks, and an attention mechanism to jointly predict crop yield and quality using multimodal inputs, including soil properties, irrigation data, microclimate conditions, and remote sensing indices. This design captures spatial patterns and temporal dependencies, while emphasizing the most influential factors, thereby enhancing both accuracy and interpretability.

The multimodal dataset at time t is defined as:

(1) $$\begin{array}{r}{X}_t=\{X_t^{soil},X_t^{irr},X_t^{clim},NDVI,EV{I}_t\}\end{array}$$

where:

Soil data: $X_t^{soil}=\{p{H}_t,E{C}_t,S{M}_t\}$ with $S{M}_t$ denoting soil moisture.

Irrigation data: $X_t^{irr}\{I_t,V_t\}$ with $I_t$ the irrigation indicator and $V_t$ the applied volume.

Microclimate data: $X_t^{clim}=\{Tem{p}_t,R{H}_t,S{H}_t\}$ where $Tem{p}_t$ air temperature, $R{H}_t$ relative humidity, and $S{H}_t$ solar radiation are.

Remote sensing indices are computed from Sentinel-2 imagery:

$NDVI=\frac{NIR-RED}{NIR+RED},EVT=$

$G\frac{NIR-RED}{NIR+C_1.RED-C_2.BLUE.+L}$ with G = 2.5; ${\text{C}}_1=6$;

${\text{C}}_2=7.5$; L = 1.

- Spatial features are extracted from vegetation indices (NDVI, EVI) using CNNs, which encode canopy vigor and crop growth conditions across multiple spatial scales:

(2) $$\begin{array}{r}{h}^{CNN}=CNN(NDVI,EVI)\end{array}$$

- Temporal dependencies from soil, irrigation, and microclimate sequences are modeled with LSTM, which effectively learns long-term seasonal patterns.

(3) $$\begin{array}{r}{h}^{LSTM}=LSTM\{X_1,X_2,\dots X_T\}\end{array}$$

- Attention mechanism assigns higher weights to influential time steps:

(4) $$\begin{array}{r}{h}^{att}={\sum }_{t=1}^T{\alpha }_t{h}_t,{\alpha }_t=\frac{\exp (e_t)}{{\sum }_{k=1}^T\exp (e_t)}\end{array}$$

This thereby improves interpretability by highlighting critical growth stages.

Concatenating outputs obtains the fused feature vector:

(5) $$\begin{array}{r}{h}^{fussion}=[h^{CNN},h^{LSTM},h^{att}]\end{array}$$

Two prediction heads are defined:

Yield prediction:

(6) $$\begin{array}{r}{\hat{Y}}_{yield}=W_y{h}^{fussion}+b_y,{\hat{Y}}_{qualiity}\in \mathbb{R}(\frac{kg}{ha})\end{array}$$

Quality prediction:

(7) $$\begin{array}{r}{\hat{Y}}_{qualiity}=\sigma (W_y{h}^{fussion}+b_q),{\hat{Y}}_{quality}\in [0,1]\end{array}$$

The multi-task loss jointly optimizes yield and quality objectives:

(8) $$\begin{array}{r}ℒ={\lambda }_{1}MSE(Y_{yield},{\hat{Y}}_{yield}){\lambda }_{2}MAE(Y_{qualiyy},{\hat{Y}}_{quality})\end{array}$$

Where λ₁ and λ₂ are weighting coefficients balancing the two objectives.

This formulation ensures that the model leverages multimodal inputs to simultaneously predict both yield and quality, a step beyond conventional yield-only approaches.

Results

This study assesses the robustness of the proposed CNN–LSTM–Attention framework across various cultivation conditions. In real-world scenarios, crop yield and quality are influenced by environmental stresses, management practices, and biological and soil-related factors, often resulting in complex and unpredictable dynamics. To address these challenges, six representative scenarios were created: (i) drought stress, (ii) excessive irrigation, (iii) heatwaves and climate anomalies, (iv) fertilizer mismanagement, (v) pest and disease outbreaks, and (vi) increasing soil salinity. These scenarios highlight critical risks that directly impact both productivity and post-harvest quality, while capturing the multidimensional interactions between soil, plants, and climate. Including these scenarios in the evaluation process ensures the model not only achieves high predictive accuracy under normal conditions but also remains stable and interpretable during stressful environments, demonstrating its practical value for precision agriculture and climate-smart farming.

Drought Stress

Water scarcity poses a significant challenge in the cultivation of rice and tomatoes. In this context, irrigation was reduced by 40% compared to the baseline, resulting in a 25% decrease in soil moisture and a 0.12 decline in NDVI.

As shown in Fig. 2, the proposed model showed higher prediction errors, with the RMSE increasing from 245 to 310 kg/ha and the MAE for quality rising from 0.18% to 0.23%. However, the CNN–LSTM–Attention framework maintained a score above 0.85, outperforming the LSTM-only baseline (0.78), thereby confirming its robustness under drought stress conditions.

Fig. 2. Predicted yield and quality performance under the drought stress scenario.

Excessive Irrigation

Over-irrigation frequently leads to root hypoxia and nutrient leaching. In this simulation, irrigation volume was increased by 50%, resulting in a 30% rise in soil moisture and a decline in EVI due to leaf yellowing. As shown in Fig. 3, the proposed model resulted in a moderate increase in yield RMSE (from 245 to 272 kg/ha) and a sharper rise in quality MAE (from 0.18% to 0.27%). The CNN–LSTM–Attention framework still outperformed the CNN-only model (MAE = 0.34%), confirming the benefit of attention in capturing stress dynamics.

Fig. 3. Model response to the excessive irrigation scenario (over-irrigation stress).

Heatwave and Climate Anomaly

Extreme temperature anomalies pose a significant threat to crop performance. In this scenario, a +5°C increase was applied for 20 consecutive days, along with a 15% drop in relative humidity. As shown in Fig. 4, the yield prediction error grew by 32% (RMSE: 245 kg/ha to 323 kg/ha), and the quality MAE increased to 0.25%. Despite this stress, the proposed framework maintained an R² above 0.83, whereas the CNN-only baseline dropped to 0.72, illustrating the benefits of sequence modeling with LSTM and attention layers.

Fig. 4. Yield and quality prediction under heatwave and climate anomaly conditions.

Fertilizer Mismanagement

The incorrect fertilizer application is a common management issue in practice. In this scenario, 30% of samples received insufficient NPK, and 20% were overfertilized (Fig. 5).

Fig. 5. Effect of fertilizer mismanagement on predicted crop yield and quality.

As presented in Fig. 5, the model exhibited a modest increase in yield RMSE (from 245 kg/ha to 268 kg/ha) and a substantial rise in quality MAE (from 0.18% to 0.25%). The CNN–LSTM–Attention approach remained more accurate than LSTM-only (MAE = 0.31%), demonstrating the benefit of integrating soil and management features with remote sensing data.

Pest and Disease Outbreak

Biotic stress significantly reduces crop vigor and spectral indices. This scenario simulated a 0.2 NDVI reduction across 25% of samples over 15 days. As illustrated in Fig. 6, the RMSE rose to 343 kg/ha, and the MAE reached 0.28%, reflecting deterioration in yield and quality. The proposed framework achieved an $R^2$ of 0.80, considerably higher than the CNN-only baseline (R² = 0.68), confirming its ability to detect biological stress factors from multimodal inputs.

Fig. 6. Impact of pest and disease outbreaks (biotic stress) on model accuracy and spectral indices.

Soil Salinity Increase

Soil salinity, often caused by seawater intrusion, is a pressing challenge in the Mekong Delta. In this scenario, EC levels were increased by 30% compared with the baseline. As illustrated in Fig. 7, the proposed CNN–LSTM–Attention framework maintains high accuracy and interpretability even under increased soil salinity conditions, confirming its robustness in environments affected by seawater intrusion.

Fig. 7. Predicted yield and quality under increased soil salinity conditions.

As shown in Fig. 7, yield RMSE increased to 288 kg/ha and quality MAE to 0.24%. Nevertheless, the CNN–LSTM–Attention framework outperformed the LSTM-only model (RMSE = 320 kg/ha), demonstrating improved robustness through the combination of soil and remote sensing features.

Discussion

Discussion of Results

The experimental results confirm that the proposed CNN–LSTM–Attention framework significantly improves the accuracy and interpretability of crop yield and quality prediction compared to single-source and single-model baselines. The integration of multimodal data—including soil, irrigation, microclimate, and remote sensing variables — enables the model to capture complex nonlinear interactions that traditional approaches often overlook. For example, the inclusion of NDVI and EVI indices from Sentinel-2 imagery helps represent canopy vigor and chlorophyll dynamics. At the same time, temporal features from microclimate and irrigation sequences enhance the model’s sensitivity to seasonal variations and management practices. Compared to CNN-only and LSTM-only models, the hybrid approach consistently achieved lower error metrics (RMSE reduced by 10%–15% and MAE by 8%–12%) across both rice and tomato datasets.

This performance improvement is due to the attention mechanism, which adaptively assigns higher weights to the most informative variables and growth stages. For rice, the reproductive and grain-filling periods received greater attention. In contrast, the flowering and fruit-setting stages were emphasized for tomatoes—aligning with agronomic knowledge about the physiological factors influencing yield and quality.

Additionally, the model exhibits stable behavior across different training runs and data partitions, indicating strong generalization ability. The joint optimization of yield and quality objectives promotes shared feature learning between tasks, allowing the model to leverage interdependencies between physiological growth patterns and post-harvest quality indicators. Statistical analysis further confirmed the robustness of these improvements, with all performance differences being significant at the p < 0.05 level (paired t-test).

Overall, these findings demonstrate that combining spatial, temporal, and contextual information through deep multimodal fusion offers a more comprehensive representation of crop systems. The proposed framework not only delivers higher prediction accuracy but also improves interpretability, providing actionable insights for data-driven agricultural management and decision support.

Robustness under Stress Scenarios

To evaluate the reliability of the proposed CNN–LSTM–Attention framework under adverse agricultural conditions, six representative stress scenarios were simulated: drought, over-irrigation, heatwave and climate anomaly, fertilizer mismanagement, pest and disease outbreaks, and soil salinity increase. These scenarios reflect common environmental and management challenges affecting rice and tomato production in tropical regions. The results demonstrate that the model maintains high robustness and accuracy across diverse stress types. Drought and heatwave events produced moderate increases in prediction error (RMSE rising from 245 to 310–323 kg/ha and MAE from 0.18% to 0.23%–0.25%), confirming the framework’s ability to capture nonlinear soil–climate interactions. Fertilizer mismanagement and soil salinity resulted in minor deviations (RMSE ≈ 268–288 kg/ha; MAE ≈ 0.24%–0.25%), whereas over-irrigation had the least impact on model stability. The most pronounced degradation occurred during pest and disease outbreaks, where spectral indices declined sharply (NDVI − 0.2) and prediction errors reached RMSE = 343 kg/ha and MAE = 0.28%.

Nevertheless, the framework consistently outperformed single-source baselines, maintaining an R² value greater than 0.80 even under biotic stress. Fig. 8 summarizes the overall performance across all scenarios, highlighting the model’s resilience and adaptability. The integration of soil, irrigation, climatic, and remote-sensing data—combined with the adaptive weighting of the attention mechanism—enables stable yield and quality prediction even when specific modalities are degraded or missing, underscoring the system’s practical applicability for climate-smart agriculture.

Fig. 8. Comparison of model performance across six stress scenarios (RMSE and MAE).

Model Interpretability and Practical

Interpretability is a critical factor for applying deep learning models in precision agriculture, as it bridges the gap between data-driven predictions and agronomic decision-making. The proposed CNN–LSTM–Attention framework not only achieves high predictive performance but also provides insights into the underlying factors influencing yield and quality outcomes. Through the attention mechanism, the model identifies key temporal segments and influential input variables that contribute most to prediction accuracy. For instance, in rice datasets, the reproductive and ripening stages exhibited the highest attention weights, reflecting their decisive impact on grain filling and final yield. Similarly, for tomato crops, flowering and fruit-setting stages were emphasized, aligning with known physiological dependencies between environmental stress, nutrient uptake, and fruit quality formation.

Visualization of the attention weights revealed that soil moisture, temperature, and vegetation indices (NDVI and EVI) were among the most dominant predictors under normal and stress conditions. This interpretability allows agronomists to better understand which management factors—such as irrigation timing, fertilizer balance, or pest control—have the greatest influence on crop performance. By linking predictive outputs to physical processes, the model facilitates trust and transparency, making it suitable for integration into agricultural decision-support systems. From a practical standpoint, the proposed framework can be embedded within IoT-based farm management platforms to enable real-time monitoring and adaptive control. For example, sensor data streams can be processed on-site or in the cloud to generate short-term forecasts of yield and quality, guiding resource allocation such as water and fertilizer usage. Furthermore, the model’s modular architecture allows extension to other crops or regions by retraining with localized datasets, supporting scalable deployment for climate-smart agriculture.

Overall, the interpretability of the CNN–LSTM–Attention model transforms deep learning from a “black box” into a transparent and actionable analytical tool, bridging the gap between artificial intelligence and agronomic expertise while fostering sustainable, data-driven crop management.

Limitations and Future Work

Despite the promising performance of the proposed CNN–LSTM–Attention framework, several limitations should be acknowledged. First, the model’s generalization capability has not been validated across a broader range of crops and regions beyond rice and tomato. Second, the dataset size remains limited, and the use of simulated stress scenarios, while useful for controlled evaluation, may not fully capture real-world variability in field conditions. Third, the computational cost of training multimodal deep learning architectures is relatively high, which may hinder large-scale or real-time deployment in low-resource environments.

Future research should focus on expanding datasets across seasons and regions, exploring lightweight Transformer-based architectures for on-device inference, and integrating uncertainty quantification to enhance reliability in practical applications.

Overall, the study highlights the potential of advanced deep learning architectures to support climate-resilient agriculture. By simultaneously optimizing yield and quality, the framework tackles both economic and nutritional goals—areas often neglected in traditional yield-focused prediction systems.

Conclusions

This study proposed a deep learning–based multimodal data integration framework that combines CNNs, LSTMs, and an attention mechanism to jointly predict crop yield and quality. Experiments on rice and tomato datasets under normal and stress conditions demonstrated that the framework achieves superior accuracy and interpretability compared with single-source and baseline methods. By fusing soil, irrigation, climate, and remote sensing data, the model significantly improves predictive performance. At the same time, the attention mechanism highlights critical growth stages and stress factors that support informed agricultural decisions. The approach also proved robust under six stress scenarios, including drought, excessive irrigation, heatwave, fertilizer mismanagement, pests/diseases, and soil salinity. Moreover, the joint optimization of yield and quality enables a more comprehensive assessment of crop performance, aligning with the dual objectives of productivity and food quality. Future work will extend this framework to additional crops and larger geographical regions, integrate Transformer-based architectures for enhanced temporal modeling, and deploy real-time decision support systems to advance precision agriculture.

Acknowledgment

The authors gratefully acknowledge the support of the University of Transport and Communications.

Conflict of Interest

The authors declare no competing financial interests or personal relationships that could have influenced the work reported in this paper.