1 Introduction

In the current field of natural language processing, English Text Generation technology is becoming increasingly important. Firstly, this technology not only enhances machines' understanding and generation of language but also advances automated content creation. For example, it plays a crucial role in news reporting, advertising copy, and literary creation. Secondly, with the continuous development of artificial intelligence technology, English Text Generation not only provides more natural and fluent communication but also meets personalized and context-specific needs, thereby improving user experience. Overall, this technology not only helps to improve the performance of language models but also holds broad application prospects in fields such as education and entertainment. Therefore, research into this technology is of significant practical importance and far-reaching impact.

Traditional methods for English text generation primarily rely on symbolic AI and knowledge representation. During this phase, expert systems, as a classic technology, generate text by utilizing predefined rules and knowledge bases. The main advantage of this approach is its ability to provide high-precision semantic processing, ensuring that the generated text adheres to specific knowledge and rules (Liu A. et al., 2021). Another method is rule-based text generation, which relies on a systematic set of language rules to ensure that the generated text is consistent and standardized in grammar and structure (Gašpar et al., 2023). Additionally, manual feature extraction is a commonly used technique, where features are manually selected and defined to drive text generation, allowing the model to focus on key language features and improve the quality of the generated text (Wang et al., 2020). These methods have distinct advantages in their respective application domains, such as high control, good interpretability, and strong structural capabilities. However, they also have certain shortcomings. For example, expert systems and rule-based methods often lack flexibility when dealing with complex and dynamic language environments. Although manual feature extraction can capture important features, it often struggles to adapt to language changes and diversity. Therefore, these traditional methods need further improvement and expansion to meet modern demands.

To address the shortcomings of traditional algorithms in terms of flexibility and adaptability, data-driven and machine learning-based algorithms have been widely applied in English text generation. These methods primarily generate text by automatically learning language patterns and features from large amounts of data. This approach has strong adaptive capabilities, allowing it to handle complex language structures and diverse expressions (Zeng, 2016). For example, decision tree-based algorithms effectively handle classification and regression problems by recursively partitioning datasets to form a series of rules. Random forest-based methods further enhance text generation stability and accuracy by constructing an ensemble model of multiple decision trees, demonstrating exceptional performance, particularly in handling high-dimensional data (Jalal et al., 2022). Additionally, the multilayer perceptron, as a type of feedforward neural network, captures complex relationships and deep features in language through the nonlinear combination of multiple hidden layers, generating more natural and fluent text (Sewunetie and Kovács, 2022). However, these methods have the drawbacks of high training complexity and a strong dependence on large-scale data, and they often exhibit insufficient generalization performance when dealing with extreme or rare language patterns.

To address the shortcomings of statistical and machine learning algorithms in feature extraction and model generalization, deep learning-based algorithms have been widely applied in English text generation. These methods primarily generate more natural and high-quality text by automatically learning complex language features and patterns through deep neural networks. This approach has significant advantages, such as the ability to handle large amounts of unstructured data, capture complex dependencies in language, and generate highly coherent and contextually appropriate content. For instance, Convolutional Neural Networks (CNNs) effectively process structural information in sentences or paragraphs by extracting local features of the text (Uchendu et al., 2020). Generative Adversarial Networks (GANs), through adversarial training between a generator and a discriminator, can generate content that closely resembles real text, enhancing the diversity and creativity of text generation (Chang et al., 2023). The Transformer model, with its self-attention mechanism, significantly improves the efficiency and accuracy of text generation, particularly excelling in the generation of long texts (Phan et al., 2022). The attention mechanism further strengthens the model's ability to capture contextual information, making the generated text more coherent and semantically consistent (Liu Y. et al., 2021). However, these methods have drawbacks, such as high model complexity, significant computational resource demands, and insufficient robustness when handling rare or unseen data.

To address the challenges posed by deep learning methods in English Text Generation, such as high model complexity, significant computational resource demands, and insufficient robustness when handling rare or unseen data, we propose a method named ETG-ALtrans. This method is based on an improved ALBEF (Align before Fuse) model and is applied to English writing guidance and error correction technology assisted by a multimodal robot. The traditional ALBEF model primarily aligns and fuses visual and linguistic information to handle multimodal tasks but faces limitations in complex language generation and semantic understanding. To overcome these issues, we optimized the ALBEF model to enhance its ability to capture contextual information in text generation while reducing its dependency on computational resources. ETG-ALtrans integrates multimodal information such as text, images, and speech to provide comprehensive English writing guidance. It effectively identifies and corrects grammatical and semantic errors in writing and generates more natural and fluent text based on context. Additionally, our method demonstrates stronger robustness when dealing with rare and unseen language patterns, improving the model's adaptability in diverse application scenarios. Experimental validation shows that ETG-ALtrans outperforms on multiple metrics, offering new insights into the development of English writing guidance technology.

• ETG-ALtrans introduces an improved ALBEF model, which combines multi-modal information to improve the comprehensive understanding and generation capabilities of text and visual content.

• This method is adaptable to multiple scenarios, efficiently handles complex writing tasks, has strong versatility, and is suitable for a variety of English writing and error correction scenarios.

• Experimental results show that ETG-ALtrans is significantly better than traditional methods in accuracy, fluency and grammatical standardization, improving the overall effect of English writing guidance and error correction.

2 Related work 2.1 Text generation

Text generation technology is a key research area in natural language processing (NLP), aiming to automatically generate natural language text that adheres to grammatical, semantic, and contextual requirements. Early text generation techniques relied primarily on template or rule-based methods. While these methods performed well in specific scenarios, they lacked flexibility and contextual understanding, making them less suitable for complex language generation tasks (Lin et al., 2024b). With the advent of statistical language models, particularly n-gram models, text generation gradually shifted toward data-driven approaches. In recent years, neural networks, especially Recurrent Neural Networks (RNNs) and Long Short-Term Memory networks (LSTMs), have played a significant role in text generation. These models can capture sequential information in text, resulting in more fluent and coherent sentences. However, these models also face challenges in handling long-range dependencies (Wang et al., 2019b). The introduction of Transformer models has brought a breakthrough in text generation technology. The self-attention mechanism of Transformers can better handle long-range dependencies and significantly improve the quality and efficiency of text generation. Transformer-based pre-trained models, such as the GPT series and BERT, have become mainstream in the field of text generation. These models, through large-scale pre-training and fine-tuning, can generate high-quality text for various tasks (Yuan et al., 2021). Notably, GPT models are widely used in dialogue systems, content creation, code generation, and other areas due to their exceptional generation capabilities. However, text generation still faces challenges such as controllability, diversity, coherence, and reducing bias and ethical issues. Future research directions may include more efficient generation models, better model interpretability, and real-time quality assessment and control of generated content.

2.2 Convolutional neural networks

Convolutional Neural Networks (CNNs) have become a core technology in computer vision since their breakthrough in image recognition tasks in 2012. CNNs are characterized by local connections, shared weights, and pooling operations, which give them strong feature extraction capabilities for handling two-dimensional data like images. Beyond image recognition, CNNs are widely applied in other visual tasks such as object detection, image segmentation, and image generation (Lin et al., 2024a). For example, in object detection, Faster R-CNN significantly improves detection speed and accuracy by introducing a Region Proposal Network (RPN). In image segmentation, architectures like U-Net and SegNet achieve fine-grained semantic segmentation by classifying each pixel in the image (Wang et al., 2019a). In addition to computer vision, CNNs are increasingly applied in other fields. In natural language processing, CNNs are used for text classification, sentiment analysis, and more. By converting text into matrix form, CNNs can capture local features of text and achieve efficient classification. In bioinformatics, CNNs are used for analyzing gene sequences and predicting protein structures, effectively identifying important patterns and features in biological sequence data. Furthermore, CNNs are applied in signal processing and time-series analysis, where convolution operations on one-dimensional or multidimensional data help analyze complex signals effectively (Fishel and Loeb, 2012). Despite the strong performance of CNNs across various fields, there are some limitations, such as reliance on large amounts of labeled data and the need for fine-tuning model structures and parameters. Future research directions may include more efficient model architectures, semi-supervised or unsupervised learning methods to reduce labeling requirements, and model optimization in low-computation resource environments (De Angelis et al., 2023).

2.3 Multimodal technology

Multimodal technology refers to the processing and understanding of information by combining different types of data (e.g., text, images, audio, video). With the diversification of data forms and advancements in computational capabilities, the importance of multimodal technology in artificial intelligence has increasingly been recognized. Early multimodal technologies focused on simple feature fusion and joint modeling, such as concatenating or averaging image and text features to achieve multimodal information integration. However, these methods often struggled to capture complex relationships between different modalities, leading to poor performance in handling multimodal data (Wang et al., 2016). Recent advancements in deep learning have significantly progressed multimodal technology. Neural network-based multimodal models, such as those combining Convolutional Neural Networks (CNNs) for image processing, Recurrent Neural Networks (RNNs) for text processing, and fully connected layers for fusion, have become mainstream. These models effectively integrate multimodal information while maintaining the independence of each modality's features, thereby improving overall task performance. The introduction of Transformer models has further advanced multimodal technology, achieving breakthroughs in handling long-range dependencies and cross-modal alignment. Models like ALBEF (Align Before Fuse) enhance the complementarity and synergy of multimodal information by aligning modalities before fusion (Fishel and Loeb, 2012). Multimodal technology has found extensive applications in various scenarios, such as image-text retrieval, cross-modal translation, and video description generation. In healthcare, multimodal technology combines medical images and text reports for more accurate disease diagnosis and treatment recommendations. In autonomous driving, multimodal technology integrates data from cameras, radar, and LiDAR to enhance environmental perception and decision-making capabilities (Qushem et al., 2021). However, multimodal technology still faces challenges such as data heterogeneity, modality misalignment, and modality weight allocation. Future developments may include more effective multimodal alignment and fusion strategies, more interpretable and robust multimodal models, and efficient deployment and optimization of multimodal systems in practical applications.

3 Methodology 3.1 Overview of our network

This research introduces an innovative optimization technique specifically designed for multimodal tasks, built upon the foundation of the ALBEF (Align Before Fuse) framework (as shown in Figure 1). It aims to refine the process to better cater to the requirements of English writing guidance and correction. To address this limitation, the paper proposes a novel set of training objectives that leverage convex functions. This novel method allows the text generation model to prioritize generating high-probability outputs without the necessity of accurately estimating the complete data distribution. Consequently, the model becomes more proficient in capturing high-probability outputs, thereby enhancing the accuracy and overall quality of the generated text. This optimization method not only improves the generative capabilities of the model but also significantly enhances its performance in practical applications, especially in tasks that require high-precision text generation and language correction. For the image encoding process, the research utilizes VGG19 as the foundational model. VGG19 is renowned for its exceptional feature extraction capabilities and straightforward yet effective structural design, making it an ideal choice for image processing in multimodal tasks. The convolutional layer architecture of VGG19 enables it to effectively capture hierarchical features in images, which can be efficiently transferred to other tasks within multimodal settings. Moreover, VGG19's streamlined design and relatively few parameters reduce computational resource demands and minimize the risk of overfitting. As a result, employing VGG19 as the image encoder not only enhances the model's stability and performance but also ensures reliable support for the efficient operation of the entire multimodal task.

Figure 1. Structure of ETG-ALtrans Net. In existing methods, video features are extracted through a visual encoder, and then passed through an error correction module to generate prediction results, which are ultimately used to generate English text. In the ETG-ALtrans method, video features are processed by a visual encoder and text features are processed by a language encoder. Then, the salient frame extraction module is used to select key frames, and the error correction module is used to generate English text. In contrast, the ETG-ALtrans method introduces the step of salient frame extraction, which improves the accuracy of generated text.

Implementation Process of the Method: In the proposed method, the overall process is divided into two main parts: text generation and image encoding, corresponding to the text editor and image encoder in the ALBEF framework. On the text editor side, we first improve the traditional MLE training method. Specifically, the training process no longer relies solely on MLE but introduces a new training objective based on convex functions. During the training phase, we designed a convex loss function that can focus the model's attention on the output with the highest generation probability. By optimizing this loss function, the model is more likely to generate text that is highly relevant to the context and adheres to linguistic rules, especially in scenarios requiring correction and assistance in English writing. This improvement makes the model more targeted during the generation phase, enhancing the quality and practicality of the generated text. On the image encoder side, a pre-trained VGG19 model is used as the base. VGG19 extracts image features through its multi-layer convolutional structure, which are then input into the ALBEF framework for alignment and fusion with text features. To ensure that the image encoder can effectively adapt to the multimodal tasks in this paper, VGG19 retains its original feature extraction capabilities during training while further optimizing to make its feature representation more accurate and representative. Through this process, the image encoder provides high-quality image feature inputs for multimodal tasks, ensuring that the model is efficient and accurate in handling multimodal data. Ultimately, the text generator and image encoder work together within the ALBEF framework to optimize the processing of multimodal tasks, thereby improving the overall performance of the model.

ALBEF as a foundational framework: While ALBEF serves as the base framework for aligning and fusing multimodal information, our model introduces crucial modifications, especially in how we handle the text generation and correction tasks. ALBEF primarily focuses on alignment and fusion of visual and textual information. In contrast, our contribution lies in developing an enhanced text editing mechanism that leverages this multimodal alignment for more effective and contextually appropriate English writing guidance.

Novel text editing framework with improved loss functions: One of our key contributions is the development of a unified framework that is compatible with various loss function configurations. We designed this framework to support more advanced learning objectives by incorporating convex functions into the loss formulation. The introduction of convex-based composite loss functions offers significant advantages, particularly for error correction and language assistance tasks, where high-precision outputs are essential. This allows the model to better focus on generating high-probability target outputs, resulting in more natural, contextually accurate text generation, which is crucial for English writing guidance and error correction. Optimization of the text generation process: Beyond simply relying on Maximum Likelihood Estimation (MLE), we propose a new objective function based on convex optimization, which allows the model to be more targeted in generating high-quality text. By incorporating these new loss functions, the model becomes more capable of producing coherent and semantically consistent text, especially in complex linguistic scenarios. This is a major enhancement over existing methods that primarily use traditional MLE for text generation. VGG19 for image encoding: On the image encoding side, we leverage VGG19 due to its proven feature extraction capabilities, and the features learned by VGG19 can be effectively transferred to other tasks, such as multimodal alignment in writing guidance. Its simplicity and robust design ensure reduced computational resource demands and minimize overfitting, which is critical when integrating visual information into text correction tasks.

Reinforcement learning for dynamic correction: The introduction of RL further distinguishes our approach. The RL mechanism enables the model to adaptively adjust its error correction strategy dynamically, optimizing the text generation and correction process as it learns from its feedback. This makes the model more flexible and responsive, especially in real-world writing scenarios where error patterns and context vary significantly. The ability to self-adjust allows the system to cater to different writing styles and needs more effectively, making it highly adaptable across diverse use cases. While our model builds on the ALBEF framework for multimodal information processing, the innovations introduced—especially in the areas of text editing through advanced loss functions, improved text generation, and the use of reinforcement learning—represent a significant departure from existing methods. These contributions collectively result in a more flexible, accurate, and adaptive system for English writing guidance and error correction.

3.2 Improved text encoder

In this section, we investigate various loss functions that can be utilized in the context of English language assistance and error rectification models (as shown in Figure 2). The goal is to overcome the limitations associated with Maximum Likelihood Estimation (MLE) (Shafiq et al., 2023). Initially, we present a unified framework that is compatible with different loss function configurations. Subsequently, we examine the advantages of incorporating convex functions as components of loss within this framework. Lastly, we propose the development of composite loss functions grounded in convex function principles, tailored to practical use cases in English language assistance and error rectification.

Figure 2. The structure of BERT. The structure of the Transformer layer and the adapter layer is shown. The adapter module enhances the task specialization ability of the model through up and down projection and non-linear activation while maintaining high efficiency.

To maintain clarity in the notation, the conditioning context is omitted from probability expressions. The actual data distribution is indicated by Ptrue(X), while the model's distribution prediction is denoted as Qmodel(X). The theoretical findings remain valid for both conditional and unconditional cases.

We begin by introducing a generalized learning framework specifically designed for English language assistance and error rectification, defined by the following loss function:

LG(R)=-?X~Ptrue(X)[G(Rmodel(X))],    (1)

where G represents a generalized function applied to the predicted probability Rmodel(X). The function G must adhere to the following fundamental conditions: (1) The domain of G should be within the interval (0, 1]; (2) G should be smooth and allow gradient computation; and (3) G should be a monotonically increasing function within (0, 1] to encourage the model to generate the optimal output for each sample.

Under the proposed framework, Maximum Likelihood Estimation (MLE) can be seen as a specific case where the function G is chosen as the natural logarithm, which is an increasing and smooth function over the domain (0, 1]. To extend the framework, one can introduce a weighted sum of two loss functions:

Ltotal(R)=γ·LH1(R)+δ·LH2(R),    (2)

where γ and δ are the weights assigned to each loss term, and H1 and H2 represent different convex functions, contributing to the composite loss.

To proceed with our analysis, we first define some key assumptions:

Premise 1 (Enumerability of the sample set): The set of possible outcomes, denoted here by X, is enumerable, which permits the systematic listing of all potential outcomes. Notably, X may either be a finite or an infinite set.

Premise 2 (Uniqueness of sample probabilities): The true data distribution, denoted by Ptrue, allocates distinct probabilities to each individual sample, allowing these samples to be ordered in a strictly descending sequence according to their respective probabilities.

Premise 1 is particularly relevant for applications in English writing support, where the inherent discreteness of text data becomes evident. Given a countable sample space and probabilities forming a dense subset of real numbers, it is plausible to assume that the probabilities assigned to each sample are unique. Although Premise 2 is not strictly required, omitting it would introduce many edge cases, complicating further analysis. Therefore, to maintain simplicity, we will assume that both Premise 1 and Premise 2 are satisfied, and samples are arranged such that Ptrue(X1) > Ptrue(X2) > ⋯ > Ptrue(Xm). With the sample space X being countable, the loss function can be expressed as:

LH(R)=-∑i=1|X|Ptrue(Xi)·H(Rmodel(Xi)).    (3)

The main goal within this framework is to analyze the probability distribution R that the model is likely to predict when employing the loss function LH. We denote Roptimal as the optimal distribution that minimizes the loss LH, reflecting the anticipated performance of the model. If LH allows multiple optimal distributions, Roptimal represents any one of these distributions. This choice does not limit the generality of our results, as the subsequent discussion is applicable to all optimal distributions. While the optimal distribution for the logarithmic loss Llog corresponds to the data distribution Ptrue, the following theorem reveals a general property of optimal distributions under other loss functions. Given that the samples are sorted in decreasing order of probability in the data distribution, Ptrue(X1) > Ptrue(X2) > ⋯ > Ptrue(Xm), any arbitrary function H preserving this order implies Roptimal(X1) ≥ Roptimal(X2) ≥ ⋯ ≥ Roptimal(Xm).

3.2.1 Loss function

In tasks that require high precision and deterministic results, such as English writing assistance and error correction, it is beneficial for the model to converge to an optimal distribution that is more concentrated than the original data distribution. This section demonstrates that using convex functions as the foundation for the learning criterion can lead to such a focused outcome. Traditional loss functions that rely on log-probability tend to be concave, which results in diminishing gradient effects as probabilities increase. This characteristic limits the model's ability to allocate high predictive probabilities to individual samples, as the incremental benefits decrease with higher probabilities. However, if the guiding function is convex, the model is more likely to converge to a more sharply concentrated distribution. The following theorem supports this observation by proving that when the function is convex, the optimal distribution transforms into a highly peaked distribution.

Theorem 2: Assume G is a monotonically increasing convex function within the interval (0, 1]. Then, the optimal distribution Roptimal is a one-hot distribution, where Roptimal(X1) = 1 and Roptimal(Xj) = 0 for all j > 1.

The concentrated nature of this optimal distribution is particularly advantageous for models dedicated to tasks such as English writing guidance, where outputs need to be precise and deterministic. For autoregressive models, this characteristic obviates the need for computationally expensive decoding methods like beam search, especially when the model's distribution is nearly one-hot. On the other hand, models that do not follow an autoregressive pattern may encounter reduced performance with traditional loss functions since they are less adept at mimicking the data distribution. However, achieving a highly concentrated optimal distribution is within the reach of these models, enabling the production of superior outputs.

Despite this, the direct implementation of convex function-based loss in training models for English writing guidance and error correction introduces a substantial obstacle, which limits its effectiveness. Specifically, when the predicted probability R(X) approaches zero, the gradient of the parameter R becomes extremely small, causing the training process to be inefficient. The gradient of R can be expressed as:

∂LG(R)∂R=-?X~Ptrue(X)[G′(Rmodel(X))·∂Rmodel(X)∂R],    (4)

where the historical dependence of R(X) has been excluded for simplicity. This equation demonstrates that the gradient is directly proportional to the probability R(X). In text generation and error correction tasks, the probability R(X), which is often derived from the probabilities of individual tokens, frequently results in R(X) being quite small, particularly when the model is still in the early phases of training.

To address this challenge, the derivative G′(R(X)) must theoretically approach infinity as R(X) approaches zero. For instance, the log-probability function has a derivative of 1R(X), effectively neutralizing the small R(X) by ensuring that G′(R(X))·R(X)=1. However, when dealing with a convex function H(R(X)) where the derivative increases with R(X), it becomes crucial that the gradient does not diminish as R(X) nears zero. This situation results in an extremely small gradient for the parameter R during training, creating a significant challenge for the practical application of convex function-based loss.

∂LH(R)∂R=-?X~Ptrue(X)[H′(Rmodel(X))·Rmodel(X)·(∑t=1T∂log(Rmodel(Xt))∂R)],    (5)

where this equation reflects how the gradient, dependent on the probability R(X), becomes challenging to manage as it approaches zero during training. This poses a significant hurdle in utilizing convex function-based loss in practice.

3.2.2 Practical applications

The preceding theoretical exploration highlights the benefits of composing functions. Now, the focus shifts toward practical implementation, where we provide examples of loss functions derived from convex composition. In English writing guidance tasks, the loss function typically emerges from a combination of several components, often integrating a term for length normalization. This results in a loss function of the form H(R(X))=log(R(X))L, where L represents the length of the sentence. Frequently used convex functions that increase over the interval (−∞, 0] include the exponential function E(R)=en·R, where n≥0, and the power function P(R)=-(-R)m, where 0 ≤ m ≤ 1. By composing these functions with H(R(X)), we obtain the following loss formulations:

These actions allow the model to select the optimal strategy based on the current text state to improve the generated text quality.

Our policy π(A|S) defines the probability distribution of selecting action A given the state S. At each step, the model chooses an action A based on the current state S to generate or modify text. The policy is updated using the policy gradient method to improve the quality of the generated text. The policy update follows the formula:

∇θJ(θ)=?πθ[∇θlogπθ(A|S)·R(S,A)]    (10)

where θ are the parameters of the policy, and R(S, A) represents the reward obtained for taking action A in state S. The gradient is estimated using Monte Carlo sampling, and the policy is optimized via gradient ascent.

To guide the model toward generating high-quality text, we design a multi-faceted reward function. This function considers not only grammatical correctness but also coherence and alignment with visual features. The reward function R(S, A) is calculated as a weighted sum of these factors:

R(S,A)=w1·Rgrammar+w2·Rcoherence+w3·Rvisual    (11)

where Rgrammar measures grammatical correctness, Rcoherence evaluates text coherence, and Rvisual assesses consistency between the generated text and visual content. The weights w1, w2, w3 balance the contributions of these factors.

For training the RL model, the following steps are followed to generate experience: 1. The model starts from an initial state S0 and generates an initial text sequence based on the multimodal inputs. 2. At each time step, the model selects an action At according to the current state St and policy π, generating or modifying the text. 3. After each step, the model receives a reward R(St, At) based on the generated result and transitions to the next state St+1. 4. The process continues until a complete text is generated, and the model accumulates rewards based on the quality of the final text.

Through these simulated experiences, the model gradually improves its strategy in multimodal writing environments, leading to text that is more grammatically correct and contextually consistent. The introduction of reinforcement learning significantly enhances the system's flexibility and adaptability. The RL mechanism allows the model to adapt correction strategies in different writing tasks, significantly improving the quality of text generation. Moreover, the multi-step decision-making capability of RL enables the model to maintain coherence and accuracy in handling long texts, particularly in multimodal scenarios where both visual and linguistic information are integrated for text optimization. Experimental results show that the RL-based model outperforms traditional rule-based systems in grammar correction and writing guidance tasks, and demonstrates superior accuracy and robustness when handling complex multimodal information.

4 Experiment 4.1 Datasets

This study used the CC12M Dataset (Changpinyo et al., 2021), MS COCO Dataset (Tong and Wu, 2023), RefCOCO Dataset (Chen et al., 2020), and VG-Cap Dataset (Ye and Kovashka, 2021) to validate the effectiveness of the multimodal robot-assisted English writing guidance and error correction technology. The CC12M Dataset provides large-scale image-text alignment data, which aids the model in learning and adapting to diverse visual and linguistic scenarios. The MS COCO Dataset contains rich image and annotation data, with high-quality semantic information supporting the model's text generation and comprehension capabilities in complex visual environments. The RefCOCO Dataset focuses on target referencing and description within specific image contexts, allowing the model to handle referential relationships more accurately and enhancing contextual understanding. The VG-Cap Dataset offers detailed image description data, further boosting the model's text generation abilities. These datasets complement each other, and through training on diverse scenes and tasks, ensure the model's robustness and practicality in various application environments, laying a solid foundation for improving the effectiveness of English writing guidance and error correction.

4.2 Experimental details

To comprehensively evaluate the effectiveness of the multimodal robot-assisted English writing guidance and error correction technology based on VGG19-ALBEF and reinforcement learning, we have designed a series of experiments, including metric comparison experiments and ablation experiments. The experiments will focus on comparing the performance of different methods across various metrics. Here are the details of the experimental design and implementation process. Firstly, in the metric comparison experiments, we will compare three different models: the traditional rule-based method, the statistical language model method, and our proposed multimodal method based on VGG19-ALBEF and reinforcement learning. Each model will be trained and tested on the same training and validation sets to ensure fairness and comparability. The training set includes 100,000 pairs of images and text from the CC12M, MS COCO, RefCOCO, and VG-Cap datasets, while the validation set consists of 20,000 pairs. These datasets are preprocessed and divided into training, validation, and test sets, with the training set making up 70% of the total data, the validation set 15%, and the test set 15%. We use TensorFlow 2.0 as the training framework, with the Adam optimizer, a learning rate of 0.001, a batch size of 64, and 50 training epochs. For each model, we record training time (in seconds) and inference time (in milliseconds), and calculate performance metrics such as model parameters (in millions), FLOPs (in billions of floating-point operations), accuracy, AUC, recall, and F1 score based on results from the test set (as is shown in Algorithm 1).