SYSTEM AND METHOD FOR AUTOMATED GRADE GENERATING USING A GENERATIVE ARTIFICIAL INTELLIGENCE (AI)

Abstract

Embodiments herein provide a method for automated grade generating using a generative artificial intelligence (AI). The method includes (i) obtaining a response an assessment in real-time from a user using a response obtaining device, (ii) enabling a generative trained machine learning model, (iii) pre-processing the response by removing a noise and irrelevant information, (iv) evaluating a score by analyzing the pre-processed response using an integrated method, where the integrated method combines at least one of a Bidirectional Encoder Representations from Transformers (BERT), a Siamese Neural Network, a Bidirectional Long Short-Term Memory (Bi-LSTM) networks, a keywords similarity technique, and a length of the answer method, (v) adjusting the evaluated score by comparing similar metrics between the evaluated score and a standard score, (vi) generating a grade based on the adjusted score along with providing feedback. FIG. 3

Specification

Description:BACKGROUND
Technical Field
[0001] The embodiments herein relate to the field of generative artificial intelligence, and more specifically to a system and method for automated evaluating responses to assessments using a generative artificial intelligence.
Description of the Related Art
[0002] Response evaluation serves as a pivotal technique inartificial intelligence. This becomes particularly crucial in the realm of educational assessment applications, healthcare applications, etc. Artificial Intelligence (AI) is an emerging solution for automated response evaluation, leveraging their ability to analyze the response and generate a score that is both realistic and diverse.. The artificial intelligence (AI) operates through a Large Language Model (LLM) that analyzes responses and generates an optimal match score
[0003] However, in the existing response evaluation system, score generation involves capturing responses during assessments from input devices. The existing response evaluation system primarily relies heavily on manual evaluation, which is both time-consuming and susceptible to human bias. However, challenges persist in response evaluation time due to delay.
[0004] The existing artificial intelligence (AI) is used in a counterfeit detection system, utilizing a Generative Adversarial Network (GAN) to differentiate between genuine and fake objects within images. This existing AI integrates a Large Language Model (LLM) to analyze the response to generate a score for the response. The LLM evaluates responses by calculating the probability of each token (word or sub-word) in a sequence, using statistical patterns from their training data to determine coherence and relevance. While this process is effective for generating plausible text, it lacks true understanding and context, leading to responses that may appear accurate but are often superficial or factually incorrect. The LLM can also unintentionally reinforce biases found in its training data and struggles with novel or highly specific queries, often resulting in outdated or irrelevant answers. Additionally, the scoring is computationally intense and lacks a real-world feedback mechanism, meaning models can't improve based on user satisfaction or outcomes. Existing systems for automated response evaluation generate scores based on a single expert's opinion, which may introduce bias. In response evaluation systems, AI relies heavily on accurate and consistent annotations from subject matter experts. Existing evaluation systems often lack feedback mechanisms for students who score poorly, which can hinder their learning and improvement, and human evaluators can introduce bias into the evaluation process. The current LLM model may struggle to effectively evaluate certain types of descriptive answers, particularly those that require images or mathematical proofs.
[0005] Accordingly, there remains a need to address the technical problem of response evaluation systems using a machine learning model.
SUMMARY
[0006] In view of the foregoing, embodiments herein provide a method for automated grade generating using a generative artificial intelligence (AI). The method includes (i) obtaining a response an assessment in real-time from a user using a response obtaining device, (ii) enabling a generative trained machine learning model, (iii) pre-processing the response by removing noise and irrelevant information, (iv) evaluating a score by analyzing the pre-processed response using an integrated method, where the integrated method combines at least one of a Bidirectional Encoder Representations from Transformers (BERT), a Siamese Neural Network, a Bidirectional Long Short-Term Memory (Bi-LSTM) networks, a keywords similarity technique, and a length of the answer method, (v) adjusting the evaluated score by comparing similar metrics between the evaluated score and a standard score, (vi) generating a grade based on the adjusted score along with providing feedback.
[0007] In some embodiments, the method includes training the machine learning model by providing pairs of assessment responses and their corresponding scores, and during training the machine learning acquires to map the response to the scores by recognizing patterns in the training data and repeatedly adjusting its internal parameters to reduce the error between its predictions and the actual scores.
[0008] In some embodiments, the method includes preprocessing the response to the assessment using at least one of a Unicode characters removal method, a case folding method, a lemmatization method, a tokenization method, a stopwords removal method, or a steaming method.
[0009] In some embodiments, the relevant text segments are extracted using at least one of an embedding method, a similarity matching method, or a retrieval of the relevant segment method for further evaluation of the response.
[0010] In some embodiments, the relevant text segments are divided are using a splitting method to extract primary relevant text segments to a query based on a split size, an overlap, and the split method.
[0011] In some embodiments, the method includes adjusting the evaluated score using at least one of an initial score analysis method, a normalization and scaling method, and additional criteria methods.
[0012] In some embodiments, the grade is generated for a numeric score using a score normalization method, a score mapping method, a grading scale method, a consistency check method, and a special case handling technique.
[0013] In some embodiments, the method includes a large language model (LLM), that evaluates the relevant text segments to understand the context and content of the response and also utilizes the grade to generate feedback.
[0014] In one aspect, a system for grade generating using a generative artificial intelligence (AI). The system includes a grade generating server that obtains response from a user using a response obtaining device. The grade generating server include a memory that stores a set of instructions and a processor that is configured to execute the set of instructions. The processor is configured to (i) obtain a response an assessment in real-time from a user using a response obtaining device, (ii) enable a generative trained machine learning model, (iii) pre-process the response by removing a noise and irrelevant information, (iv) evaluate a score by analyzing the pre-processed response, (v) adjust the evaluated score by comparing similar metrics between the evaluated score and a standard score, (vi) generate a grade based on the adjusted score along with providing feedback.
[0015] The method is of advantage that the method reduces the burden of manual evaluation and is efficient. The method completes the response evaluation independently, without bias toward a single expert answer taken from any text, reference book, or prepared by a human evaluator, by integrating Large Language Models (LLMs) with Generative Artificial Intelligence (Gen-AI). The method provides a feedback mechanism for students who scored very low, and assigns final marks by averaging multiple evaluations.
[0016] Further, the method is of advantage that the method considers various factors when evaluating a descriptive answer, such as keywords, complete string similarity, the order of sentences in the student's answer compared to the gold standard answer, explanations of any images related to the required answer, and the use of any necessary mathematical symbols.
[0017] These and other aspects of the embodiments herein will be better appreciated and understood when considered in conjunction with the following description and the accompanying drawings. It should be understood, however, that the following descriptions, while indicating preferred embodiments and numerous specific details thereof, are given by way of illustration and not of limitation. Many changes and modifications may be made within the scope of the embodiments herein without departing from the spirit thereof, and the embodiments herein include all such modifications.
BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The embodiments herein will be better understood from the following detailed description with reference to the drawings, in which:
[0019] FIG. 1 is a block diagram of a system for grade generating using a generative artificial intelligence (AI) according to some embodiments herein;
[0020] FIG. 2 is a block diagram a grade generating server of FIG. 1 according to some embodiments herein;
[0021] FIG. 3 is a graphical representation of different model similarity scores with ideal responses according to some embodiments herein;
[0022] FIG. 4 is a graphical representationof the comparison of different model average error percentages according to some embodiments herein;
[0023] FIGS. 5 is a flow diagram of a method for grade generating using a generative artificial intelligence (AI) according to some embodiments herein; and
[0024] FIG. 6 is a schematic diagram of a computer architecture of the server or one or more entity devices in accordance with embodiments herein.
DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS
[0025] The embodiments herein and the various features and advantageous details thereof are explained more fully with reference to the non-limiting embodiments that are illustrated in the accompanying drawings and detailed in the following description. Descriptions of well-known components and processing techniques are omitted so as to not unnecessarily obscure the embodiments herein. The examples used herein are intended merely to facilitate an understanding of ways in which the embodiments herein may be practiced and to further enable those of skill in the art to practice the embodiments herein. Accordingly, the examples should not be construed as limiting the scope of the embodiments herein.
[0026] There remains a need for a system and method for grade generating using a generative artificial intelligence (AI). Referring now to the drawings, and more particularly to FIGS. 1 to 6, where similar reference characters denote corresponding features consistently throughout the figures, there are shown preferred embodiments.
[0027] The term " Generative Trained Machine Learning Model" refers to a model designed to generate new data samples that resemble a given set of training data. Instead of just recognizing or categorizing data, these models learn the underlying patterns and structures of the input data to create similar but unique outputs. Examples of generative models include Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and autoregressive models. They are widely used in tasks such as image synthesis, text generation, and other applications where producing new, realistic data is beneficial.
[0028] The term " Generative Artificial Intelligence" refers to refers to a branch of AI focused on creating systems that can produce new, original content, such as text, images, audio, and video, that resembles human-generated material. Unlike traditional AI, which typically analyzes or classifies data, generative AI models are trained to learn underlying patterns in large datasets and generate novel outputs based on those patterns. This capability is achieved through models like Generative Adversarial Networks (GANs), Large Language Models (LLMs), and Variational Autoencoders (VAEs). Generative AI has transformative applications in fields ranging from entertainment and design to customer service and personalized content creation.
[0029] FIG. 1 is a block diagram of a system 100 for grade generating using a generative artificial intelligence (AI)according to some embodiments herein. The system 100 includes a response obtaining device 102, and a grade generating server 106. The grade generating server 106 includes a generative trained machine learning model 108. A list of devices that are capable of hosting the grade generating server 106, without limitation, may include one or more personal computers, laptops, tablet devices, smartphones, mobile communication devices, personal digital assistants, or any other such computing device. The response obtaining device 102 may be accessed by a user. The user may include but is not limited to, a student or interviewee.
[0030] The user is communicatively connected to a grade generating server 106 through a network 104 to obtain response from the response obtaining device 102. The response is the answer or reply given by the student or candidate to a question, prompt, or problem presented to them. In exams, responses reflect the individual's understanding of the subject matter, demonstrating their knowledge, problem-solving skills, or analytical abilities. In interviews, responses help assess the candidate's experience, skills, and suitability for the role or opportunity, offering insights into their personality, communication skills, and thought process. Effective responses are clear, relevant, and directly address the questions asked, showcasing the individual's preparedness and competence.
[0031] In some embodiments, the network 106 is a wireless network. In some embodiments, the network 106 is a wired network. In some embodiments, the network 106 is a combination of the wired network and the wireless network. In some embodiments, network 106 is the Internet. The grade generating server 106 includes a memory that stores a set of instructions and a processor. The processor executes the set of instructions and is configured to generate a grade based on score of the response.
[0032] The grade generating server 106 obtains response using the response obtaining device 102 associated with the user during an assessment. The grade generating server 106 pre-processes the response obtained by the user using methods such as Unicode character removal, case folding, lemmatization, tokenization, stopword removal, and stemming. The grade generating server 106 evaluates a score for the response by analyzing the pre-processed response using a trained machine learning model (ML). A machine learning model (ML) is trained on a large dataset of expert answers, carefully curated to cover a broad range of topics, writing styles, and levels of complexity. During training, the system learns patterns, structures, and features characteristic of well-written answers. This knowledge is then applied to evaluate student responses. The trained ML evaluates a score for the response using Bidirectional Encoder Representations from Transformers (BERT), a cosine similarity method, a Siamese Neural Network, a Bidirectional Long Short-Term Memory (Bi-LSTM) networks, a keywords similarity technique, and a length of the answer method.
[0033] The grade generating server 106 adjusts the evaluated score using an initial score analysis method, a normalization and scaling method, and additional criteria methods. The grade generating server 106 generates a grade along with feedback for the adjusted numeric score using a score normalization method, a score mapping method, a grading scale method, a consistency check method, and a special case handling technique. The feedback is generated using a large language model (LLM), which thoroughly analyzes the response to provide insightful, specific, and constructive guidance based on the content. The trained ML extracts relevant text segments of the response and a query associated to the response using an embedding method, a similarity matching method, or a retrieval of relevant segment method for further evaluation of the response. the relevant text segments are divided are using a splitting method to extract most relevant text segment to a query based on a split size, an overlap, and the split method.
[0034] The grade generating server 106 includes a generative trained machine learning model 108. The generative trained machine learning model 108 is a system that learns patterns and relationships from a dataset to make predictions or classifications on new, unseen data. This is crucial for automated grade generation as it ensures consistency in grading, providing a fair and standardized assessment for all students. The model enhances efficiency by rapidly evaluating large volumes of responses, making it scalable for educational institutions. Additionally, the generative trained machine learning model 108 minimizes bias, offering objective assessments and can adapt over time as new data becomes available. Furthermore, the trained models can provide detailed, personalized feedback, helping students identify areas for improvement and facilitating their learning process. generative trained machine learning model 108 Large Language Models (LLMs) along with Generative Artificial Intelligence (Gen-AI), encompassing multiple text documents on a specific subject. This enables the evaluation process to be conducted independently, free from bias toward any single expert answer obtained from texts, reference books, or prepared by human evaluators.
[0035] FIG. 2 is a block diagram of the grade generating server 106 of FIG. 1 according to some embodiments herein. The grade generating server 106 includes response obtaining module 202, a generative trained machine learning enabling module 108, a pre-processing module 204, an evaluation module 206, a score adjusting module 208, a grade generating module 210, a feedback generation module 212, and a database 200. The response obtaining module 202 receives a response to an assessment. The pre-processing module 204 pre-processes the response by removing noise and irrelevant information and subsequently applies advanced NLP techniques to comprehend the context, semantics, and structure of the response. The pre-processing module 204 utilizes a unicode character removal method, a case folding method, a lemmatization method, a tokenization method, a stopword removal method, and a stemming method. The unicode character removal method ensures that the text data is clean and standardized. The unicode character removal method can encompass various symbols, accents, special characters, and even emojis that are unnecessary for text analysis. The unicode character removal method is a character encoding standard that accommodates a wide array of characters from different languages. However, some unicode character removal methods may not be compatible with the NLP tools or algorithms employed in the system. The case folding method converts all characters in the text to lowercase to ensure uniformity. Text data can appear in different cases. The cases may be an uppercase, a lowercase, or a mixed case. By converting all characters to lowercase, the generative trained machine learning model treats words with the same meaning equally, regardless of their casing. The method prevents the model from recognizing "Apple" and "apple" as distinct words. The lemmatization method reduces words to their base or dictionary form, known as the lemma. This process is more sophisticated than stemming, which merely truncates word endings. The lemmatization method involves understanding the context and identifying the intended part of speech of a word before converting it to base form. For example, "better" is transformed into "good" through the lemmatization method. The lemmatization method aids in standardizing words that have different forms but convey the same meaning. involves splitting text into individual words or tokens. The tokenization method breaks down a sentence or text into smaller components, typically words. The tokenization method is essential as tokenization enables the system to analyze each word individually. Depending on the language or complexity of the text, tokenization may also require managing punctuation, contractions, or special symbols. The stopword removal method filters out common words that do not add meaningful content. The stopword removal method include frequently used words like "the," "is," "in," and "and," which typically lack significant meaning on their own but are essential for sentence structure. By eliminating these words, the system can concentrate on the more important terms that contribute to the overall meaning of the text, thereby reducing noise in the data. The stemming method reduces words to their root forms by stripping suffixes (and occasionally prefixes) from words. Unlike lemmatization, stemming does not take the context of the word into account and often produces non-dictionary words. The method is helpful for consolidating variations of a word into a single form, thereby simplifying the analysis. Stemming reduces words to their root forms by stripping suffixes (and occasionally prefixes) from words. Unlike lemmatization, stemming does not take the context of the word into account and often produces non-dictionary words. This method helps consolidate variations of a word into a single form, thereby simplifying the analysis.
[0036] The system is designed to teach the characteristics of high-quality responses through a comprehensive training process. The machine learning model (ML) is trained on a large dataset of expertly curated answers that represent a wide array of topics, writing styles, and levels of complexity. During the training phase, the ML learns to identify patterns, structures and features common to well-written responses, which enables it to effectively evaluate student answers. The core function of the Training Module is to train the machine learning model to map input text (response) to output labels (scores) based on observed patterns in the training data. The training process typically uses supervised learning, where the model is provided with input-output pairs consisting of response and corresponding score. The model then iteratively adjusts its internal parameters to minimize the difference between its predictions and the actual scores.
[0037] The evaluation module 206 evaluates a score of the response using a Bidirectional Encoder Representations from Transformers (BERT), a Cosine Similarity method, a Siamese Neural Network, a Bidirectional Long Short-Term Memory (Bi-LSTM) networks, a keywords similarity technique, and a length of the answer method. The BERT is utilized to capture the context and meaning of text by analyzing the relationships between words within a sentence. As a transformer-based model, BERT processes text bidirectionally, meaning it takes into account the context from both the left and right sides of each word. This bidirectional approach enables BERT to grasp nuanced meanings and relationships within the text that traditional unidirectional models may overlook. In the evaluation module 206, the BERT generates embeddings numerical representations of both expert response and student responses. These embeddings are then compared to evaluate semantic similarity. The Cosine Similarity method measures the cosine of the angle between two vectors (representing text) to determine the similarity between the two vectors. After converting the text into numerical vectors using the BERT embeddings the Cosine Similarity calculates the cosine of the angle between these vectors in multi-dimensional space. A similarity score of 1 indicates that the vectors are identical, while a score of 0 shows that they are orthogonal, or completely dissimilar. The Cosine Similarity method effectively measures the similarity between two pieces of text, regardless of their length. The Siamese Networks use twin neural networks with shared weights to assess the similarity between two inputs. The architecture consists of two identical neural networks (with shared parameters) that process the inputs separately but in parallel. The outputs from each network are then compared to generate a similarity score. The Siamese networks are especially effective for tasks requiring similarity assessment, such as comparing student answers with expert answers. During training, the network learns to differentiate between similar and dissimilar pairs, enhancing its ability to evaluate new pairs accurately. The Bidirectional Long Short-Term Memory (Bi-LSTM) network combines the Siamese network with a Bidirectional Long Short-Term Memory (Bi-LSTM) network to improve contextual understanding. The Bi-LSTM network is a type of recurrent neural network (RNN), that captures long-term dependencies in text. When integrated into a Siamese architecture, the Bi-LSTM processes text in both forward and backward directions, allowing for a deeper grasp of the context surrounding each word. The method enhances the capability to capture the semantics of complex sentences, making it especially valuable for evaluating descriptive student answers where context and meaning are essential. The Keywords similarity technique compares essential keywords between the user response and the expert response to assess content relevance. The Keyword similarity technique is a significant term or phrase that conveys a key meaning within a text. The Keyword Similarity technique identifies and matches these terms in both the user response and the expert response. A higher presence and frequency of similar keywords suggest that the user response includes relevant content. The approach is often combined with other techniques to provide a more comprehensive evaluation. The length of the answer method prioritizes content quality considering the length of the response as an indicator of thoroughness and detail. The length of the answer method compares the length of the response to the standard of the expert, where significant deviations may suggest the student has either over-explained or under-explained the topic.
[0038] The score adjusting module 208 receives the initial similarity scores calculated by the Evaluation Module 206. The scores represent the raw comparison between the user response and the expert response, using various similarity metrics. The score adjusting module 208 utilizes an initial score analysis method, a normalization and scaling method, and additional criteria methods. The initial score analysis method begins by analyzing the initial similarity scores produced by the evaluation module 206. The scores are derived from techniques such as BERT embeddings, cosine similarity, and other methods that evaluate how closely the user response aligns with the expert standard. The methods for evaluating additional criteria may include contextual relevance, answer length, difficulty level, user performance history, and outlier detection. Once these additional criteria have been taken into account, the adjusting module 208 may normalize or scale the scores to ensure they conform to the grading standards. The adjustment guarantees that all scores are comparable and that no student is unjustly advantaged or disadvantaged. Although the adjustments are automated, educators can manually intervene if they disagree with the automated score modifications. For example, teachers can adjust the score before finalization. Additionally, the system may implement threshold-based adjustments, applying automatic modifications if a similarity score falls within a specified range.
[0039] The grade generating module 210 generates the grade for the adjusted numeric score using a score normalization method, a score mapping method, a grading scale method, a consistency check method, and a special case handling technique. The score normalization method. The score normalization method normalizes the adjusted score to fit a predefined grading scale, ensuring all scores are proportionally aligned, regardless of the initial raw values before assigning the final grade. The score mapping method converts adjusted similarity scores into final grades. The score mapping method involves mapping numerical scores to a grading scale, which may include letter grades (A, B, C, etc.), percentage scores, or another standardized grading system used by the educational institution. The grading scale method is usually set by the institution or educator. For example, scores from 90-100 may correspond to an A, 80-89 to a B, and so forth. The Grading Module uses the grading scale method to translate adjusted scores into final grades. The consistency check method performs a consistency check to verify that assigned grades meet expectations, preventing anomalies like a high similarity score leading to a low grade, or the reverse. The special case handling technique includes a partial credit method and a bonus point method. The partial credit method assigns partial credit based on the extent to which the answer aligns with the expert standard. The bonus point method adds bonus points for outstanding answers or applies penalties for issues like grammatical errors or incomplete responses.
[0040] The trained ML model receives the response and associated query as input, typically in text form, to analyze and extract relevant segments using an embedding method, a similarity matching method, and a relevant segment retrieval method for further evaluation. The embeddings, which are dense, high-dimensional vectors, represent the semantic meaning of words, phrases, sentences, or even larger text segments. These embeddings enable the system to compare and retrieve related content effectively. The trained ML leverages these embeddings to convert both user response and reference content for comparison and retrieval. Once the text is transformed into embeddings, identify which parts of the user response are most relevant to the query. It searches for segments within the user response that closely match the semantic content of the query. Techniques like Cosine Similarity or Dot Product are used to measure the closeness of embeddings, with higher similarity scores indicating that a particular segment of the user response is highly relevant to the query.
[0041] The relevant text segments of the response have further divided the text into smaller, manageable parts using a splitting method based on one or more parameters. The splitting method is essential as large text segments can be challenging for the system to analyze all at once. Splitting the text enables a more detailed and focused analysis. The parameters may be a split size, an overlapping, and a split method. The split size refers to the length or size of each segment after splitting, which could be determined by the number of words, sentences, or even tokens (smaller units of text, such as individual words or sub-words). The overlapping Specifies the amount of text that overlaps between consecutive segments, helping to retain important contextual information. For instance, if a segment cuts off in the middle of a key concept, this overlap ensures the concept is carried into the next segment for continuity. The split method Sets the criteria for dividing the text, such as splitting by sentences, paragraphs, or fixed-length chunks, depending on the text's nature and evaluation requirements. The challenge in text splitting is preserving context and meaning across segments. The machine learning model ensures that dividing text does not disrupt its context, making parameters like overlap essential. By carefully managing the split, the system can maintain an understanding of each segment's relevance and meaning within the complete answer. Breaking text into smaller segments allows for easier individual analysis, enabling more precise comparisons between the user response and the expert standard. The granularity reduces computational load, facilitating faster and more efficient processing by the Evaluation Module 206.
[0042] The feedback generation module 212 utilizes a Large Language Model (LLM) to analyze segmented portions of a user response, allowing it to understand the context and key concepts within the response. The LLM employs natural language processing to evaluate the relevance and coherence of the response, generating personalized feedback that addresses specific strengths and weaknesses. By correlating this feedback with the grade from the grade generating module 210, the LLM ensures that the feedback is aligned with the overall assessment of the user performance. The iterative approach enables the user to understand the mistakes and improve future responses based on the insights provided.
[0043] FIG. 3 is an exemplary view of the different model similarity scores with ideal responses according to some embodiment herein. The applicant system offers a comprehensive evaluation of various models designed to assess descriptive answers, comparing their performance against manual evaluation to identify the most accurate technique. Starting with the BERT Text Similarity model, the research recorded an accuracy of approximately 85.4%. The implementation of Cosine Vector Similarity led to a notable improvement, reducing the average error by 3.8%. Expanding the investigation to include Siamese Text Similarity and Bi-LSTM Siamese Text Similarity models yielded significantly enhanced results, achieving accuracies of around 94.2% and 97.6%, respectively. In contrast, reliance on basic approaches such as Keyword Matching and Answer Length Similarity negatively affected precision, lowering it to approximately 79.2%.
[0044] FIG. 4 is an exemplary view of the comparison of different model average error percentages according to some embodiments herein. The application of Large Language Models (LLMs) has significantly transformed evaluation metrics. Fine-tuning the applicant system led to impressive results, reducing the average error to just 1% surplus and achieving an exceptional accuracy of 99% in assessing descriptive student answers. In comparison, all other models appeared less effective, underscoring the substantial benefits provided by LLMs.
[0045] The below table:1 presents a comparison of different model performance and manual evaluation-
Response ID Evaluator
Score BERT Similarity Cosine Similarity Siamese Text Similarity Bi-LSTM Siamese Similarity Keywords and Answer Length Applicant
1 7.0 5.5 6.5 6.7 6.8 4.5 7.3
2 8.5 7.3 7.2 7.4 8.3 6.6 8.6
3 9.0 7.9 8.1 8.6 8.8 7.4 8.9
4 7.5 5.1 6.1 7.1 7.1 5.3 7.8
5 8.0 6.9 6.7 7.3 7.8 5.8 7.9
8.0 6.54 6.92 7.42 7.76 5.92 8.1

[0046] FIGS. 5 is a flow diagram of a method for automated grade generating using a generative artificial intelligence (AI) according to some embodiments herein. At step 502, the method includes obtaining a response an assessment in real-time from a user using a response obtaining device. At step 504, the method includes enabling a generative trained machine learning model. At step 506, the method includes pre-processing the response by removing a noise and irrelevant information. At step 508, the method includes evaluating a score by analyzing the pre-processed response using an integrated method, where the integrated method combines at least one of a Bidirectional Encoder Representations from Transformers (BERT), a Siamese Neural Network, a Bidirectional Long Short-Term Memory (Bi-LSTM) networks, a keywords similarity technique, and a length of the answer method. At step 510, the method includes adjusting the evaluated score by comparing similar metrics between the evaluated score and a standard score. At step 512, the method includes generating a grade based on the adjusted score along with providing feedback.
[0047] The embodiments herein may include a computer program product configured to include a pre-configured set of instructions, which when performed, can result in actions as stated in conjunction with the methods described above. In an example, the pre-configured set of instructions can be stored on a tangible non-transitory computer readable medium or a program storage device. In an example, the tangible non-transitory computer readable medium can be configured to include the set of instructions, which when performed by a device, can cause the device to perform acts similar to the ones described here. Embodiments herein may also include tangible and/or non-transitory computer-readable storage media for carrying or having computer executable instructions or data structures stored thereon.
[0048] Generally, program modules utilized herein include routines, programs, components, data structures, objects, and the functions inherent in the design of special-purpose processors, etc. that perform particular tasks or implement particular abstract data types. Computer executable instructions, associated data structures, and program modules represent examples of the program code means for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps.
[0049] The embodiments herein can include both hardware and software elements. The embodiments that are implemented in software include but are not limited to, firmware, resident software, microcode, etc.
[0050] A data processing system suitable for storing and/or executing program code will include at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution.
[0051] Input/output (I/O) devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the system either directly or through intervening I/O controllers. Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modem, and Ethernet cards are just a few of the currently available types of network adapters.
[0052] A representative hardware environment for practicing the embodiments herein is depicted in FIG. 6, with reference to FIGS. 1 through 5. This schematic drawing illustrates a hardware configuration of a server 106 or a computer system or a computing device in accordance with the embodiments herein. The system includes at least one processing device CPU 10 that may be interconnected via system bus 14 to various devices such as a random-access memory (RAM) 15, read-only memory (ROM) 17, and an input/output (I/O) adapter 17. The I/O adapter 17 can connect to peripheral devices, such as disk units 12 and program storage devices 13 that are readable by the system. The system can read the inventive instructions on the program storage devices 13 and follow these instructions to execute the methodology of the embodiments herein. The system further includes a user interface adapter 20 that connects a keyboard 18, mouse 19, speaker 25, microphone 23, and other user interface devices such as a touch screen device (not shown) to the bus 14 to gather user input. Additionally, a communication adapter 21 connects the bus 14 to a data processing network 42, and a display adapter 22 connects the bus 14 to a display device 24, which provides a graphical user interface (GUI) 30 of the output data in accordance with the embodiments herein, or which may be embodied as an output device such as a monitor, printer, or transmitter, for example.
[0053] The foregoing description of the specific embodiments will so fully reveal the general nature of the embodiments herein that others can, by applying current knowledge, readily modify and/or adapt for various applications such specific embodiments without departing from the generic concept, and, therefore, such adaptations and modifications should and are intended to be comprehended within the meaning and range of equivalents of the disclosed embodiments. It is to be understood that the phraseology or terminology employed herein is for the purpose of description and not of limitation. Therefore, while the embodiments herein have been described in terms of preferred embodiments, those skilled in the art will recognize that the embodiments herein can be practiced with modification within the spirit and scope of the appended claims.
, Claims: I/We Claim:
1. A method for automated grade generating using a generative artificial intelligence (AI), wherein the method comprises:
obtaining a response an assessment in real-time from a user using a response obtaining device;
enabling a generative trained machine learning model;
pre-processing the response by removing a noise and irrelevant information;
evaluating a score by analysing the pre-processed response using an integrated method, wherein the integrated method combines at least one of a Bidirectional Encoder Representations from Transformers (BERT), a Siamese Neural Network, a Bidirectional Long Short-Term Memory (Bi-LSTM) networks, a keywords similarity technique, and a length of the answer method;
adjusting the evaluated score by comparing similar metrics between the evaluated score and a standard score; and
generating a grade based on the adjusted score along with providing feedback.

2. The method as claimed in claim 1, wherein the machine learning model is trained by providing pairs of assessment responses and their corresponding scores, and during training the machine learning acquire to map the response to the scores by recognizing patterns in the training data and repeatedly adjusting its internal parameters to reduce the error between its predictions and the actual scores.

3. The method as claimed in claim 1, wherein the method comprising preprocessing the response to the assessment using at least one of a unicode characters removal method, a case folding method, a lemmatization method, a tokenization method, a stopwords removal method, or a steaming method.

4. The method as claimed in claim 1, wherein the relevant text segments are extracted using at least one of embedding method, a similarity matching method, or a retrieval of relevant segment method for further evaluation of the response.
5. The method as claimed in claim 1, wherein the relevant text segments are divided are using a splitting method to extract primary relevant text segment to a query based on a split size, an overlap, and the split method.

6. The method as claimed in claim 1, wherein the evaluated score is adjusted using at least one of an initial score analysis method, a normalization and scaling method, and additional criteria methods.

7. The method as claimed in claim 1, wherein the grade is generated for a numeric score using a score normalization method, a score mapping method, a grading scale method, a consistency check method, and a special case handling technique.

8. The method as claimed in claim 1, wherein a large language model (LLM) evaluates the relevant text segments to understand the context and content of the response and utilizes the grade to generate feedback.

9. An automated system for grade generating using a generative artificial intelligence (AI), wherein the system comprises:
a grade generating server that comprises:
a memory that includes a generative trained machine-learning (ML) model: and
a processor that executes the trained ML model and is configured to:
obtain a response to an assessment in real-time provided from a user using a response obtaining device.
enable a generative trained machine learning model;
pre-process the response by removing a noise and irrelevant information;
evaluate a score by analysing the pre-processed response using an integrated method, wherein the integrated method combines at least one of a Bidirectional Encoder Representations from Transformers (BERT), a Siamese Neural Network, a Bidirectional Long Short-Term Memory (Bi-LSTM) networks, a keywords similarity technique, and a length of the answer method;
adjust the evaluated score by comparing similar metrics between the evaluated score and a standard score; and generate a grade based on the adjusted score along with providing feedback.


Dated this November 19, 2024

Arjun Karthik Bala
(IN/PA 1021)
Agent for Applicant

Documents