From ba12c676c9ae54019f98ac6e6e328d7181dc2b6e Mon Sep 17 00:00:00 2001 From: Vsevolod Stakhov Date: Wed, 3 Jul 2024 15:34:02 +0100 Subject: [PATCH 01/15] Start post about GPT --- _posts/2024-07-03-gpt.md | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 _posts/2024-07-03-gpt.md diff --git a/_posts/2024-07-03-gpt.md b/_posts/2024-07-03-gpt.md new file mode 100644 index 000000000..5bb588d6e --- /dev/null +++ b/_posts/2024-07-03-gpt.md @@ -0,0 +1,38 @@ +--- +layout: post +title: "Rspamd and GPT integration" +categories: misc +--- + +## Preface + +Historically, we had only Bayes as the text classification method. Bayes is still quite a powerful statistical method that can grant quite a decent performance with enough of learning. There are two main disadvantages of the Bayes: + +* It requires a lot of balanced and well designed learning +* It cannot work when there is not enough confidence, especially for high variety of spam + +Using of a large language models (LLM) can help address both of the issues, as such models can perform deep intorspection with some sort of "understanding" of the context. However, LLM models require quite a lot of computational resources (usually GPU), so it is not practical to scan all email via them. It is also quite beneficial to run these models separately from a scan engine to avoid resources racing. + +## Rspamd GPT plugin + +In Rspamd 3.9, I have tried to play with OpenAI API and decide if it is useful in spam filtering or not. Here are some basic ideas behind this plugin: + +* We select a displayed text part, extract text from it and ask GPT API for probability of it to be spam +* We also add some more information from the message, such as subject, displayed from, url information +* Then we ask GPT to make JSON output as we can parse JSON and we cannot parse human readable GPT output (in general) +* We exclude some specific symbols from being scan on GPT (e.g. `BAYES_SPAM`, `FUZZY_DENIED` as well as `REPLY` and other similar symbols) +* We also exclude apparent spam and ham from the checks + +The former two points is done to reduce GPT load for something that we already know about and there is nothing that GPT can add in the evaluation. We also use GPT as one of the classifiers, meaning that we do not rely on GPT evaluation only. + +## Evaluation results + +TBD + +## Pricing concerns + +TBD + +## Conclusions + +TBD \ No newline at end of file From c0cacd9b44f0f4858d60f8163d2cf04879b59808 Mon Sep 17 00:00:00 2001 From: moisseev Date: Thu, 4 Jul 2024 15:31:15 +0300 Subject: [PATCH 02/15] Improve GPT integration post --- _posts/2024-07-03-gpt.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/_posts/2024-07-03-gpt.md b/_posts/2024-07-03-gpt.md index 5bb588d6e..dddc1f70b 100644 --- a/_posts/2024-07-03-gpt.md +++ b/_posts/2024-07-03-gpt.md @@ -1,38 +1,38 @@ --- layout: post -title: "Rspamd and GPT integration" +title: "Integrating Rspamd with GPT" categories: misc --- ## Preface -Historically, we had only Bayes as the text classification method. Bayes is still quite a powerful statistical method that can grant quite a decent performance with enough of learning. There are two main disadvantages of the Bayes: +Historically, our only text classification method has been Bayes, a powerful statistical method that performs well with sufficient training. However, Bayes has its limitations: -* It requires a lot of balanced and well designed learning -* It cannot work when there is not enough confidence, especially for high variety of spam +* It requires thorough and well-balanced training +* It cannot work with low confidence levels, especially when dealing with a wide variety of spam -Using of a large language models (LLM) can help address both of the issues, as such models can perform deep intorspection with some sort of "understanding" of the context. However, LLM models require quite a lot of computational resources (usually GPU), so it is not practical to scan all email via them. It is also quite beneficial to run these models separately from a scan engine to avoid resources racing. +Large Language Models (LLMs) offer promising solutions to these challenges. These models can perform deep introspection with some sort of contextual "understanding". However, their high computational demands (typically requiring GPUs) make scanning all emails impractical. Separating LLM execution from the scanning engine mitigates resource competition. ## Rspamd GPT plugin -In Rspamd 3.9, I have tried to play with OpenAI API and decide if it is useful in spam filtering or not. Here are some basic ideas behind this plugin: +In Rspamd 3.9, I have tried to integrate the OpenAI GPT API for spam filtering and assess its usefulness. Here are some basic ideas behind this plugin: -* We select a displayed text part, extract text from it and ask GPT API for probability of it to be spam -* We also add some more information from the message, such as subject, displayed from, url information -* Then we ask GPT to make JSON output as we can parse JSON and we cannot parse human readable GPT output (in general) -* We exclude some specific symbols from being scan on GPT (e.g. `BAYES_SPAM`, `FUZZY_DENIED` as well as `REPLY` and other similar symbols) -* We also exclude apparent spam and ham from the checks +* The selected displayed text part is extracted and submitted to the GPT API for spam probability assessment +* Additional message details such as Subject, Display From, and URLs are also included in the assessment +* Then we ask GPT to provide results in JSON format since human-readable GPT output cannot be parsed (in general) +* Some specific symbols (`BAYES_SPAM`, `FUZZY_DENIED`, `REPLY`, etc.) are excluded from the GPT scan +* Obvious spam and ham are also excluded from the GPT evaluation -The former two points is done to reduce GPT load for something that we already know about and there is nothing that GPT can add in the evaluation. We also use GPT as one of the classifiers, meaning that we do not rely on GPT evaluation only. +The former two points reduce the GPT workload for something that is already known, where GPT cannot add any value in the evaluation. We also use GPT as one of the classifiers, meaning that we do not rely solely on GPT evaluation. ## Evaluation results TBD -## Pricing concerns +## Pricing considerations TBD ## Conclusions -TBD \ No newline at end of file +TBD From 5dc33767e01bbdd6430e7fe0a24d8b1feb05d4a7 Mon Sep 17 00:00:00 2001 From: Vsevolod Stakhov Date: Thu, 4 Jul 2024 15:28:56 +0100 Subject: [PATCH 03/15] Add some more content --- _posts/2024-07-03-gpt.md | 78 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 77 insertions(+), 1 deletion(-) diff --git a/_posts/2024-07-03-gpt.md b/_posts/2024-07-03-gpt.md index dddc1f70b..700ebbaa2 100644 --- a/_posts/2024-07-03-gpt.md +++ b/_posts/2024-07-03-gpt.md @@ -27,7 +27,83 @@ The former two points reduce the GPT workload for something that is already know ## Evaluation results -TBD +To evaluate the performance of the GPT based classifier, we have created an utility `rspamadm classifier_test` that can evaluate both supervised and unsupervised classifiers. + +* It splits spam and ham samples into two sets: training and validation +* For supervised classifiers it uses training set to learn classifier +* Then for both supervised and unsupervised classifiers it uses validation set to check the performace of the classifier + +For example, Bayes engine, learned on a decent training corpus has the following result: + +~~~ +$ rspamadm classifier_test --ham /ham --spam /spam --cv-fraction 0.3 + +Spam: 348 train files, 815 cv files; ham: 754 train files, 1762 cv files +Start learn spam, 348 messages, 10 connections +Start learn ham, 754 messages, 10 connections +Learning done: 348 spam messages in 1.61 seconds, 754 ham messages in 3.88 seconds +Start cross validation, 2577 messages, 10 connections +Metric Value +------------------------------ +True Positives 735 +False Positives 22 +True Negatives 1717 +False Negatives 49 +Accuracy 0.97 +Precision 0.97 +Recall 0.94 +F1 Score 0.95 +Classified (%) 97.90 +Elapsed time (seconds) 12.71 +~~~ + +These results are quite good, but they assume that the classifier is properly and decently trained. For the case when we have a fresh system or high variability of emails it might be close to impossible to gather somehow decent statistics. + +GPT engine can be treated in this case as unsupervised learning algorithm: we don't really 'learn' it on emails, we assume that LLM models have enough "understanding" of the language to distinguish spam and ham. We also do not provide raw emails to them, only text data. + +So here are some results from different models: + +### OpenAI GPT3.5-turbo + +~~~ +Metric Value +------------------------------ +True Positives 129 +False Positives 35 +True Negatives 263 +False Negatives 69 +Accuracy 0.79 +Precision 0.79 +Recall 0.65 +F1 Score 0.71 +Classified (%) 95.20 +Elapsed time (seconds) 318.91 +~~~ + +This model is quite cheap and can be used as the baseline. The results were obtained from the corpus with low quality samples, so there are quite a lot of FP/FN in the results. + +### OpenAI GPT4-o + +~~~ +Metric Value +------------------------------ +True Positives 178 +False Positives 25 +True Negatives 257 +False Negatives 9 +Accuracy 0.93 +Precision 0.88 +Recall 0.95 +F1 Score 0.91 +Classified (%) 90.02 +Elapsed time (seconds) 279.08 +~~~ + +This model is expensive and very advanced but it can still be used, for example, for the low traffic personal email. The results were also obtained from the corpus with low quality samples, but the error rate is significantly lower than with GPT3.5. + +## Plugin design + +TODO ## Pricing considerations From a608ea79a1c04bcb15f9cad5463aaacd717bd1f0 Mon Sep 17 00:00:00 2001 From: moisseev Date: Thu, 4 Jul 2024 19:39:01 +0300 Subject: [PATCH 04/15] Improve "Evaluation results" section --- _posts/2024-07-03-gpt.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/_posts/2024-07-03-gpt.md b/_posts/2024-07-03-gpt.md index 700ebbaa2..7309995a0 100644 --- a/_posts/2024-07-03-gpt.md +++ b/_posts/2024-07-03-gpt.md @@ -27,13 +27,13 @@ The former two points reduce the GPT workload for something that is already know ## Evaluation results -To evaluate the performance of the GPT based classifier, we have created an utility `rspamadm classifier_test` that can evaluate both supervised and unsupervised classifiers. +To evaluate the performance of the GPT-based classifier, we developed the `rspamadm classifier_test` utility, capable of evaluating both supervised and unsupervised classifiers: -* It splits spam and ham samples into two sets: training and validation -* For supervised classifiers it uses training set to learn classifier -* Then for both supervised and unsupervised classifiers it uses validation set to check the performace of the classifier +* It divides spam and ham samples into separate training and validation sets +* For supervised classifiers, it uses the training set to train the classifier +* Then both supervised and unsupervised classifiers are evaluated using the validation set to measure their performance -For example, Bayes engine, learned on a decent training corpus has the following result: +For example, the Bayes engine, trained on a robust corpus, demonstrates the following results: ~~~ $ rspamadm classifier_test --ham /ham --spam /spam --cv-fraction 0.3 @@ -57,13 +57,13 @@ Classified (%) 97.90 Elapsed time (seconds) 12.71 ~~~ -These results are quite good, but they assume that the classifier is properly and decently trained. For the case when we have a fresh system or high variability of emails it might be close to impossible to gather somehow decent statistics. +These results are impressive but assume the classifier is properly and decently trained. In scenarios involving a fresh system or high variability in emails, gathering reliable statistics might be challenging. -GPT engine can be treated in this case as unsupervised learning algorithm: we don't really 'learn' it on emails, we assume that LLM models have enough "understanding" of the language to distinguish spam and ham. We also do not provide raw emails to them, only text data. +In contrast, the GPT engine operates as an unsupervised learning algorithm. We assume that LLM models have enough "understanding" of the language to distinguish spam and ham without direct training on emails. Moreover, we provide only text data, not raw email content. -So here are some results from different models: +Below are the results from different GPT models: -### OpenAI GPT3.5-turbo +### OpenAI GPT-3.5 Turbo ~~~ Metric Value @@ -80,9 +80,9 @@ Classified (%) 95.20 Elapsed time (seconds) 318.91 ~~~ -This model is quite cheap and can be used as the baseline. The results were obtained from the corpus with low quality samples, so there are quite a lot of FP/FN in the results. +This model is cost-effective and can be used as a baseline. The results were obtained from a low-quality sample corpus, resulting in high false positives and negatives. -### OpenAI GPT4-o +### OpenAI GPT-4o ~~~ Metric Value @@ -99,7 +99,7 @@ Classified (%) 90.02 Elapsed time (seconds) 279.08 ~~~ -This model is expensive and very advanced but it can still be used, for example, for the low traffic personal email. The results were also obtained from the corpus with low quality samples, but the error rate is significantly lower than with GPT3.5. +Despite its high cost, this advanced model is suitable, for example, for low-traffic personal email. It demonstrates significantly lower error rates compared to GPT-3.5, even with a similar low-quality sample corpus. ## Plugin design From 33a7a71f9dbab51322f81c97ea1c3a085e376380 Mon Sep 17 00:00:00 2001 From: Vsevolod Stakhov Date: Sat, 6 Jul 2024 13:39:31 +0100 Subject: [PATCH 05/15] Add more text --- _posts/2024-07-03-gpt.md | 57 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 56 insertions(+), 1 deletion(-) diff --git a/_posts/2024-07-03-gpt.md b/_posts/2024-07-03-gpt.md index 7309995a0..d67d56d76 100644 --- a/_posts/2024-07-03-gpt.md +++ b/_posts/2024-07-03-gpt.md @@ -101,9 +101,64 @@ Elapsed time (seconds) 279.08 Despite its high cost, this advanced model is suitable, for example, for low-traffic personal email. It demonstrates significantly lower error rates compared to GPT-3.5, even with a similar low-quality sample corpus. +### Using GPT to learn Bayes + +Another interesting option is to use GPT to supervise Bayes engine learning. In this case, we get the best of two worlds: GPT can work without training and Bayes can afterwards catch up and perform instead of GPT (or at least act as a cost saving option). + +So we have tested GPT training Bayes and compared efficiency using the same methodics. + +GPT results: + +~~~ +Metric Value +------------------------------ +True Positives 128 +False Positives 13 +True Negatives 301 +False Negatives 68 +Accuracy 0.84 +Precision 0.91 +Recall 0.65 +F1 Score 0.76 +Classified (%) 97.89 +Elapsed time (seconds) 341.77 +~~~ + +And here are the results from the Bayes trained by GPT in the previous test iteration: + +~~~ +Metric Value +------------------------------ +True Positives 19 +False Positives 43 +True Negatives 269 +False Negatives 9 +Accuracy 0.85 +Precision 0.31 +Recall 0.68 +F1 Score 0.42 +Classified (%) 65.26 +Elapsed time (seconds) 29.18 +~~~ + +As we can see, Bayes is still not very confident in classification and has somehow more FP than GPT. On the other hand, this could be further improved by autolearning and by selecting a better corpus to test (our corpus has, indeed, a lot of HAM emails that look like spam even for a human) + ## Plugin design -TODO +GPT plugin has the following operation logic: + +* It selects messages that qualify several pre-checks: + - they must not have any symbols from the `excluded` set (e.g. Fuzzy/Bayes spam/Whitelists) + - they must not be apparent ham or spam (e.g. with reject action or with no action with high negative score) + - they should have enough text tokens in the meaningful displayed part +* If a message satisfies the checks, Rspamd selects the displayed part (e.g. HTML) and uses the following content to send to GPT: + - text part content as one line string (honoring limits if necessary) + - message's subject + - displayed from + - some information about URLs (e.g. domains) +* This data is also merged with a prompt to GPT that orders to evaluate a probability of such an email to be spam, and output the result as JSON (other output types can sometimes allow GPT to use a human readable text that is very difficult to parse) +* After all these operations, a corresponding symbol with confidence score is inserted +* If autolearning is enabled, then Rspamd also learns the supervised classifier (meaning Bayes) ## Pricing considerations From 5012de06c9f07e91bf93bd0dd7a5f687137be0ef Mon Sep 17 00:00:00 2001 From: Vsevolod Stakhov Date: Sat, 6 Jul 2024 15:00:27 +0100 Subject: [PATCH 06/15] More text --- _posts/2024-07-03-gpt.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/_posts/2024-07-03-gpt.md b/_posts/2024-07-03-gpt.md index d67d56d76..526d233c1 100644 --- a/_posts/2024-07-03-gpt.md +++ b/_posts/2024-07-03-gpt.md @@ -162,7 +162,9 @@ GPT plugin has the following operation logic: ## Pricing considerations -TBD +OpenAI provides API for the requests and it costs some money (there is no free tier so far). However, if you plan to use it for a personal email or if you just want to train your Bayes without manual classification, GPT might be a good option to consider. As a concrete example, for my personal email (that is quite a loaded one), the cost of gpt-3.5 is around $0.05 per day (for like 100k tokens). + +For the large scale email systems, it is probably better to get some other LLM (e.g. llama) and use it internally on a system with some GPU power. The existing plugin is designed to work with other LLM types without significant modifications. ## Conclusions From 272760429afedec8167c94e218819dea7c77570a Mon Sep 17 00:00:00 2001 From: Vsevolod Stakhov Date: Sat, 6 Jul 2024 15:27:10 +0100 Subject: [PATCH 07/15] Add some final content --- _posts/2024-07-03-gpt.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/_posts/2024-07-03-gpt.md b/_posts/2024-07-03-gpt.md index 526d233c1..11ad37164 100644 --- a/_posts/2024-07-03-gpt.md +++ b/_posts/2024-07-03-gpt.md @@ -160,12 +160,12 @@ GPT plugin has the following operation logic: * After all these operations, a corresponding symbol with confidence score is inserted * If autolearning is enabled, then Rspamd also learns the supervised classifier (meaning Bayes) -## Pricing considerations +## Pricing considerations and conclusions OpenAI provides API for the requests and it costs some money (there is no free tier so far). However, if you plan to use it for a personal email or if you just want to train your Bayes without manual classification, GPT might be a good option to consider. As a concrete example, for my personal email (that is quite a loaded one), the cost of gpt-3.5 is around $0.05 per day (for like 100k tokens). -For the large scale email systems, it is probably better to get some other LLM (e.g. llama) and use it internally on a system with some GPU power. The existing plugin is designed to work with other LLM types without significant modifications. +For the large scale email systems, it is probably better to get some other LLM (e.g. llama) and use it internally on a system with some GPU power. The existing plugin is designed to work with other LLM types without significant modifications. This method has also another advantage by providing more privacy of your data as you do not send the content of your emails to some 3-rd party service, such as OpenAI (however, they claim that their models are not learned on API requests). -## Conclusions +Despite of not being 100% accurate, GPT plugin provides efficiency that is roughly about the efficiency of a human filtered email. We plan to work further on accuracy improvements by adding more metadata to GPT engine trying to stay efficient in terms of tokens usage. There are other plans for better usage of the LLM knowlege in Rspamd, for example, for better fine-grained classification. -TBD +GPT plugin will be available from Rspamd 3.9, and you still need to apply for API key from OpenAI (and invest some funds there) to use the ChatGPT. \ No newline at end of file From da2dc87e13d68497adea862ba03d237674145f8c Mon Sep 17 00:00:00 2001 From: Vsevolod Stakhov Date: Sat, 6 Jul 2024 15:30:35 +0100 Subject: [PATCH 08/15] Add link on the index page --- index.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/index.md b/index.md index 9f2d51fda..11c8af99b 100644 --- a/index.md +++ b/index.md @@ -12,6 +12,16 @@ title: Rspamd spam filtering system
+
+
+

Study: Rspamd and GPT integration

+ 2024-07-03 +

We have studied the efficiency of the new GPT plugin to access LLM (large language models) intelligence.

+
+
+ Learn more  +
+

News: Rspamd 3.8.4 has been released

From cf2fb9f5349c1579496793fdd8a7963e54cd12cc Mon Sep 17 00:00:00 2001 From: Vsevolod Stakhov Date: Sat, 6 Jul 2024 15:45:03 +0100 Subject: [PATCH 09/15] Add documentation for the plugin --- doc/modules/gpt.md | 101 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 101 insertions(+) create mode 100644 doc/modules/gpt.md diff --git a/doc/modules/gpt.md b/doc/modules/gpt.md new file mode 100644 index 000000000..e4533ac37 --- /dev/null +++ b/doc/modules/gpt.md @@ -0,0 +1,101 @@ +# Rspamd GPT Plugin + +In Rspamd 3.9, ther is a new plugin to use OpenAI GPT API for spam filtering. Here are some basic ideas behind this plugin: + +* The selected displayed text part is extracted and submitted to the GPT API for spam probability assessment +* Additional message details such as Subject, Display From, and URLs are also included in the assessment +* Then we ask GPT to provide results in JSON format since human-readable GPT output cannot be parsed (in general) +* Some specific symbols (`BAYES_SPAM`, `FUZZY_DENIED`, `REPLY`, etc.) are excluded from the GPT scan +* Obvious spam and ham are also excluded from the GPT evaluation + +The former two points reduce the GPT workload for something that is already known, where GPT cannot add any value in the evaluation. We also use GPT as one of the classifiers, meaning that we do not rely solely on GPT evaluation. + +You can read more about this plugin in this [document](https://rspamd.com/misc/2024/07/03/gpt.html). + +## Configuration Options + +```ini +gpt { + # Supported types: openai + type = "openai"; + + # Your key to access the API + api_key = "xxx"; + + # Model name + model = "gpt-3.5-turbo"; + + # Maximum tokens to generate + max_tokens = 1000; + + # Temperature for sampling + temperature = 0.7; + + # Top p for sampling + top_p = 0.9; + + # Timeout for requests + timeout = 10s; + + # Prompt for the model (use default if not set) + prompt = "xxx"; + + # Custom condition (lua function) + condition = "xxx"; + + # Autolearn if gpt classified + autolearn = true; + + # Reply conversion (lua code) + reply_conversion = "xxx"; + + # URL for the API + url = "https://api.openai.com/v1/chat/completions"; +} +``` + +### Description of Configuration Options + +- **type**: Specifies the type of GPT model to use. Currently, only "openai" is supported. + +- **api_key**: The API key required to access the OpenAI services. Replace "xxx" with your actual API key. + +- **model**: Defines the model name to be used. For example, "gpt-3.5-turbo". + +- **max_tokens**: The maximum number of tokens to generate in the response. This controls the length of the generated text. + +- **temperature**: A sampling parameter that controls the creativity of the generated text. Values range from 0 to 1, with higher values resulting in more creative outputs. The default is 0.7. + +- **top_p**: Another sampling parameter known as nucleus sampling. It determines the cumulative probability for token selection. The default value is 0.9. + +- **timeout**: Sets the maximum time to wait for a response from the API. Specified in seconds (e.g., 10s). + +- **prompt**: The initial text prompt to guide the model's generation. If not set, a default prompt will be used. + +- **condition**: A custom Lua function to define specific conditions for using the plugin. + +- **autolearn**: When set to true, the plugin will automatically learn and adjust based on the model's classifications. + +- **reply_conversion**: Custom Lua code to convert the model's reply into a specific format or handle it in a certain way. The default one implies JSON output of the GPT model. + +- **url**: The endpoint for the OpenAI API. The default is "https://api.openai.com/v1/chat/completions". + +## Example Configuration + +Here is an example configuration with the fields filled in: + +~~~ucl +gpt { + type = "openai"; + api_key = "your_api_key_here"; + model = "gpt-3.5-turbo"; + max_tokens = 500; + temperature = 0.6; + top_p = 0.8; + timeout = 15s; +} +~~~ + +## Conclusion + +The Rspamd GPT Plugin provides powerful text processing capabilities using OpenAI's GPT models. By configuring the options above, users can tailor the plugin to meet specific needs and enhance their email processing workflows. \ No newline at end of file From b026a6fea32bbb4c7f3e54f2f4530eced8997eb3 Mon Sep 17 00:00:00 2001 From: moisseev Date: Sun, 7 Jul 2024 11:36:30 +0300 Subject: [PATCH 10/15] Improve the remaining sections --- _posts/2024-07-03-gpt.md | 42 ++++++++++++++++++++-------------------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/_posts/2024-07-03-gpt.md b/_posts/2024-07-03-gpt.md index 11ad37164..8484c83dd 100644 --- a/_posts/2024-07-03-gpt.md +++ b/_posts/2024-07-03-gpt.md @@ -18,7 +18,7 @@ Large Language Models (LLMs) offer promising solutions to these challenges. Thes In Rspamd 3.9, I have tried to integrate the OpenAI GPT API for spam filtering and assess its usefulness. Here are some basic ideas behind this plugin: * The selected displayed text part is extracted and submitted to the GPT API for spam probability assessment -* Additional message details such as Subject, Display From, and URLs are also included in the assessment +* Additional message details such as Subject, displayed From, and URLs are also included in the assessment * Then we ask GPT to provide results in JSON format since human-readable GPT output cannot be parsed (in general) * Some specific symbols (`BAYES_SPAM`, `FUZZY_DENIED`, `REPLY`, etc.) are excluded from the GPT scan * Obvious spam and ham are also excluded from the GPT evaluation @@ -101,11 +101,11 @@ Elapsed time (seconds) 279.08 Despite its high cost, this advanced model is suitable, for example, for low-traffic personal email. It demonstrates significantly lower error rates compared to GPT-3.5, even with a similar low-quality sample corpus. -### Using GPT to learn Bayes +### Using GPT to train Bayes classifier -Another interesting option is to use GPT to supervise Bayes engine learning. In this case, we get the best of two worlds: GPT can work without training and Bayes can afterwards catch up and perform instead of GPT (or at least act as a cost saving option). +Another interesting approach involves using GPT to supervise Bayes engine training. In this case, we benefit from the best of both worlds: GPT can operate without training, while Bayes can catch up afterward and perform instead of GPT (or at least serve as a cost-saving alternative). -So we have tested GPT training Bayes and compared efficiency using the same methodics. +So we tested GPT training Bayes and compared efficiency using the same methodologies. GPT results: @@ -124,7 +124,7 @@ Classified (%) 97.89 Elapsed time (seconds) 341.77 ~~~ -And here are the results from the Bayes trained by GPT in the previous test iteration: +Bayes classifier results (trained by GPT in the previous test iteration): ~~~ Metric Value @@ -141,31 +141,31 @@ Classified (%) 65.26 Elapsed time (seconds) 29.18 ~~~ -As we can see, Bayes is still not very confident in classification and has somehow more FP than GPT. On the other hand, this could be further improved by autolearning and by selecting a better corpus to test (our corpus has, indeed, a lot of HAM emails that look like spam even for a human) +Bayes still exhibits uncertainty in classification, with more false positives than GPT. Improvement could be achieved through autolearning and by refining the corpus used for testing (our corpus contains many ham emails that look like spam even for human evaluators). ## Plugin design -GPT plugin has the following operation logic: +The GPT plugin operates as follows: * It selects messages that qualify several pre-checks: - - they must not have any symbols from the `excluded` set (e.g. Fuzzy/Bayes spam/Whitelists) - - they must not be apparent ham or spam (e.g. with reject action or with no action with high negative score) + - they must not contain any symbols from the `excluded` set (e.g. Fuzzy/Bayes spam/Whitelists) + - they must not clearly appear as ham or spam (e.g. with reject action or no action with a high negative score) - they should have enough text tokens in the meaningful displayed part -* If a message satisfies the checks, Rspamd selects the displayed part (e.g. HTML) and uses the following content to send to GPT: - - text part content as one line string (honoring limits if necessary) - - message's subject - - displayed from - - some information about URLs (e.g. domains) -* This data is also merged with a prompt to GPT that orders to evaluate a probability of such an email to be spam, and output the result as JSON (other output types can sometimes allow GPT to use a human readable text that is very difficult to parse) -* After all these operations, a corresponding symbol with confidence score is inserted -* If autolearning is enabled, then Rspamd also learns the supervised classifier (meaning Bayes) +* If a message satisfies these checks, Rspamd selects the displayed part (e.g. HTML) and sends the following content to GPT: + - text part content as a single-line string (honoring limits if necessary) + - message subject + - displayed From + - some details about URLs (e.g. domains) +* This data is merged with a prompt to GPT requesting an evaluation of the email's spam probability, with the output returned in JSON format (other output types may sometimes allow GPT to provide human-readable text that is very difficult to parse) +* After these steps, a corresponding symbol with a confidence score is inserted +* With autolearning enabled, Rspamd also trains the supervised classifier (Bayes) ## Pricing considerations and conclusions -OpenAI provides API for the requests and it costs some money (there is no free tier so far). However, if you plan to use it for a personal email or if you just want to train your Bayes without manual classification, GPT might be a good option to consider. As a concrete example, for my personal email (that is quite a loaded one), the cost of gpt-3.5 is around $0.05 per day (for like 100k tokens). +OpenAI provides an API for these requests, incurring costs (currently no free tier available). However, for personal email usage or automated Bayes training without manual intervention, GPT presents a viable option. For instance, processing a substantial volume of personal emails with GPT-3.5 costs approximately $0.05 daily (for about 100k tokens). -For the large scale email systems, it is probably better to get some other LLM (e.g. llama) and use it internally on a system with some GPU power. The existing plugin is designed to work with other LLM types without significant modifications. This method has also another advantage by providing more privacy of your data as you do not send the content of your emails to some 3-rd party service, such as OpenAI (however, they claim that their models are not learned on API requests). +For large-scale email systems, it may be preferable to use another LLM (e.g. llama) internally on a GPU-powered platform. The current plugin is designed to integrate with different LLM types without significant modifications. This approach also enhances data privacy by avoiding sending email content to a third-party service (though OpenAI claims their models do not learn from API requests). -Despite of not being 100% accurate, GPT plugin provides efficiency that is roughly about the efficiency of a human filtered email. We plan to work further on accuracy improvements by adding more metadata to GPT engine trying to stay efficient in terms of tokens usage. There are other plans for better usage of the LLM knowlege in Rspamd, for example, for better fine-grained classification. +Despite not achieving 100% accuracy, the GPT plugin demonstrates efficiency comparable to human-filtered email. Future enhancements will focus on improving accuracy through additional metadata integration into the GPT engine, while optimizing token usage efficiency. There are also plans to better utilize LLM knowledge in Rspamd, particularly for better fine-grained classification. -GPT plugin will be available from Rspamd 3.9, and you still need to apply for API key from OpenAI (and invest some funds there) to use the ChatGPT. \ No newline at end of file +The GPT plugin will be available starting from Rspamd 3.9, requiring an OpenAI API key and financial commitment for accessing ChatGPT services. From 17cdf406f79b2269a08cce98d0f92112264b6bf8 Mon Sep 17 00:00:00 2001 From: moisseev Date: Sun, 7 Jul 2024 11:53:24 +0300 Subject: [PATCH 11/15] Fix local links --- index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/index.md b/index.md index 11c8af99b..52d1f139b 100644 --- a/index.md +++ b/index.md @@ -19,7 +19,7 @@ title: Rspamd spam filtering system

We have studied the efficiency of the new GPT plugin to access LLM (large language models) intelligence.

- Learn more  + Learn more 
@@ -29,7 +29,7 @@ title: Rspamd spam filtering system

New stable release is now available.

- Learn more  + Learn more 
From d8ce9c1d328d2519dc2b3d008accbcab20ff3a8b Mon Sep 17 00:00:00 2001 From: moisseev Date: Sun, 7 Jul 2024 17:10:46 +0300 Subject: [PATCH 12/15] Improve plugin documentation --- _posts/2024-07-03-gpt.md | 4 ++-- doc/modules/gpt.md | 50 ++++++++++++++++++++-------------------- 2 files changed, 27 insertions(+), 27 deletions(-) diff --git a/_posts/2024-07-03-gpt.md b/_posts/2024-07-03-gpt.md index 8484c83dd..8cbf88286 100644 --- a/_posts/2024-07-03-gpt.md +++ b/_posts/2024-07-03-gpt.md @@ -15,11 +15,11 @@ Large Language Models (LLMs) offer promising solutions to these challenges. Thes ## Rspamd GPT plugin -In Rspamd 3.9, I have tried to integrate the OpenAI GPT API for spam filtering and assess its usefulness. Here are some basic ideas behind this plugin: +In Rspamd 3.9, I have tried to integrate the OpenAI GPT API for spam filtering and assess its usefulness. Here are the basic ideas behind this plugin: * The selected displayed text part is extracted and submitted to the GPT API for spam probability assessment * Additional message details such as Subject, displayed From, and URLs are also included in the assessment -* Then we ask GPT to provide results in JSON format since human-readable GPT output cannot be parsed (in general) +* Then, we ask GPT to provide results in JSON format since human-readable GPT output cannot be parsed (in general) * Some specific symbols (`BAYES_SPAM`, `FUZZY_DENIED`, `REPLY`, etc.) are excluded from the GPT scan * Obvious spam and ham are also excluded from the GPT evaluation diff --git a/doc/modules/gpt.md b/doc/modules/gpt.md index e4533ac37..d95e6ccf9 100644 --- a/doc/modules/gpt.md +++ b/doc/modules/gpt.md @@ -1,25 +1,25 @@ # Rspamd GPT Plugin -In Rspamd 3.9, ther is a new plugin to use OpenAI GPT API for spam filtering. Here are some basic ideas behind this plugin: +The Rspamd GPT Plugin, introduced in Rspamd 3.9, integrates OpenAI's GPT API to enhance spam filtering capabilities using advanced natural language processing techniques. Here are the basic ideas behind this plugin: * The selected displayed text part is extracted and submitted to the GPT API for spam probability assessment -* Additional message details such as Subject, Display From, and URLs are also included in the assessment -* Then we ask GPT to provide results in JSON format since human-readable GPT output cannot be parsed (in general) +* Additional message details such as Subject, displayed From, and URLs are also included in the assessment +* Then, we ask GPT to provide results in JSON format since human-readable GPT output cannot be parsed (in general) * Some specific symbols (`BAYES_SPAM`, `FUZZY_DENIED`, `REPLY`, etc.) are excluded from the GPT scan * Obvious spam and ham are also excluded from the GPT evaluation The former two points reduce the GPT workload for something that is already known, where GPT cannot add any value in the evaluation. We also use GPT as one of the classifiers, meaning that we do not rely solely on GPT evaluation. -You can read more about this plugin in this [document](https://rspamd.com/misc/2024/07/03/gpt.html). +For detailed information about this plugin, refer to the [blog post](https://rspamd.com/misc/2024/07/03/gpt.html). ## Configuration Options ```ini gpt { - # Supported types: openai + # Supported type: openai type = "openai"; - # Your key to access the API + # Your OpenAI API key api_key = "xxx"; # Model name @@ -40,51 +40,51 @@ gpt { # Prompt for the model (use default if not set) prompt = "xxx"; - # Custom condition (lua function) + # Custom condition (Lua function) condition = "xxx"; - # Autolearn if gpt classified + # Autolearn if GPT classified autolearn = true; - # Reply conversion (lua code) + # Reply conversion (Lua code) reply_conversion = "xxx"; - # URL for the API + # URL for the OpenAI API url = "https://api.openai.com/v1/chat/completions"; } ``` ### Description of Configuration Options -- **type**: Specifies the type of GPT model to use. Currently, only "openai" is supported. +- **type**: Specifies the GPT model type. Currently, only "openai" is supported. -- **api_key**: The API key required to access the OpenAI services. Replace "xxx" with your actual API key. +- **api_key**: Your API key for accessing OpenAI services. Replace "xxx" with your actual API key. -- **model**: Defines the model name to be used. For example, "gpt-3.5-turbo". +- **model**: Specifies the GPT model to use, such as "gpt-3.5-turbo". -- **max_tokens**: The maximum number of tokens to generate in the response. This controls the length of the generated text. +- **max_tokens**: Sets the maximum number of tokens to generate in the GPT response, controlling the length of the generated text. -- **temperature**: A sampling parameter that controls the creativity of the generated text. Values range from 0 to 1, with higher values resulting in more creative outputs. The default is 0.7. +- **temperature**: Controls the creativity of the generated text during sampling. Values range from 0 to 1, with higher values resulting in more creative outputs (default: 0.7). -- **top_p**: Another sampling parameter known as nucleus sampling. It determines the cumulative probability for token selection. The default value is 0.9. +- **top_p**: Sets the cumulative probability threshold for token selection using nucleus sampling (default: 0.9). -- **timeout**: Sets the maximum time to wait for a response from the API. Specified in seconds (e.g., 10s). +- **timeout**: Specifies the maximum wait time for API responses in seconds (e.g., 10s). -- **prompt**: The initial text prompt to guide the model's generation. If not set, a default prompt will be used. +- **prompt**: Optional initial text prompt guiding model generation. If not set, a default prompt is used. -- **condition**: A custom Lua function to define specific conditions for using the plugin. +- **condition**: Custom Lua function defining conditions for plugin usage. -- **autolearn**: When set to true, the plugin will automatically learn and adjust based on the model's classifications. +- **autolearn**: Enables automatic learning based on GPT classifications when set to true. -- **reply_conversion**: Custom Lua code to convert the model's reply into a specific format or handle it in a certain way. The default one implies JSON output of the GPT model. +- **reply_conversion**: Custom Lua code converting the model's reply into a specific format or handling it in a certain way (default: JSON output of the GPT model). -- **url**: The endpoint for the OpenAI API. The default is "https://api.openai.com/v1/chat/completions". +- **url**: Endpoint for the OpenAI API (default: "https://api.openai.com/v1/chat/completions"). ## Example Configuration Here is an example configuration with the fields filled in: -~~~ucl +```hcl gpt { type = "openai"; api_key = "your_api_key_here"; @@ -94,8 +94,8 @@ gpt { top_p = 0.8; timeout = 15s; } -~~~ +``` ## Conclusion -The Rspamd GPT Plugin provides powerful text processing capabilities using OpenAI's GPT models. By configuring the options above, users can tailor the plugin to meet specific needs and enhance their email processing workflows. \ No newline at end of file +The Rspamd GPT Plugin integrates OpenAI's GPT models into Rspamd, enhancing its spam filtering capabilities with advanced text processing techniques. By configuring the options above, users can customize the plugin to meet specific requirements, thereby enhancing the efficiency and accuracy of spam filtering within Rspamd. From 6bdb989a55b18789defa3b359bbb5b778546d427 Mon Sep 17 00:00:00 2001 From: moisseev Date: Sun, 7 Jul 2024 17:19:18 +0300 Subject: [PATCH 13/15] Add missing front matter --- doc/modules/gpt.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/modules/gpt.md b/doc/modules/gpt.md index d95e6ccf9..fbe387c52 100644 --- a/doc/modules/gpt.md +++ b/doc/modules/gpt.md @@ -1,3 +1,7 @@ +--- +layout: doc +title: GPT Plugin +--- # Rspamd GPT Plugin The Rspamd GPT Plugin, introduced in Rspamd 3.9, integrates OpenAI's GPT API to enhance spam filtering capabilities using advanced natural language processing techniques. Here are the basic ideas behind this plugin: From 0755727c4eb63081917dfc4d4348b3faf30fc2ad Mon Sep 17 00:00:00 2001 From: moisseev Date: Sun, 7 Jul 2024 17:26:10 +0300 Subject: [PATCH 14/15] Fix syntax highlighting --- doc/modules/gpt.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/modules/gpt.md b/doc/modules/gpt.md index fbe387c52..4b246485d 100644 --- a/doc/modules/gpt.md +++ b/doc/modules/gpt.md @@ -18,7 +18,7 @@ For detailed information about this plugin, refer to the [blog post](https://rsp ## Configuration Options -```ini +```hcl gpt { # Supported type: openai type = "openai"; From 0236a480b774e0761a690c9114b4084669e5aeee Mon Sep 17 00:00:00 2001 From: moisseev Date: Sun, 7 Jul 2024 17:36:00 +0300 Subject: [PATCH 15/15] Fix local link --- doc/modules/gpt.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/modules/gpt.md b/doc/modules/gpt.md index 4b246485d..145c7ad6c 100644 --- a/doc/modules/gpt.md +++ b/doc/modules/gpt.md @@ -14,7 +14,7 @@ The Rspamd GPT Plugin, introduced in Rspamd 3.9, integrates OpenAI's GPT API to The former two points reduce the GPT workload for something that is already known, where GPT cannot add any value in the evaluation. We also use GPT as one of the classifiers, meaning that we do not rely solely on GPT evaluation. -For detailed information about this plugin, refer to the [blog post](https://rspamd.com/misc/2024/07/03/gpt.html). +For detailed information about this plugin, refer to the [blog post]({{ site.baseurl }}/misc/2024/07/03/gpt.html). ## Configuration Options