Added ability to monitor PasteBin Users

.gitignore

@@ -101,3 +101,8 @@ ENV/
 .mypy_cache/
 /settings.conf
 /YaraRules/custom_keywords.yar
+/paste_history.tmp
+/settings.json
+/.idea
+/postprocess/tester.py
+.vscode/

README.md

@@ -1,16 +1,75 @@
 # PasteHunter
-Scan pastebin pastes with a collection of yara rules.
-
-# PreReqs
+PasteHunter is a python3 application that is designed to query a collection of sites that host publicliy pasted data. 
+For all the pasts it finds it scans the raw contents against a series of yara rules looking for information that can be used 
+by an org or a researcher.
+
+## Supported Inputs
+Pastehunter currently has support for the following sites:
+ - pastebin.com
+ - dumpz.org
+ - gist.github.com
+
+Support for the following sites is listed as ToDo:
+ - paste.ee
+
+## Supported Outputs
+Pastehunter supports several output modules:
+ - dump to ElasticSearch DB (default)
+ - email sending over SMTP
+ - dump to JSON file
+ - dump to CSV file
+
+### SMTP
+Multiple recipients can be specified, with different rulesets each.
+It's possible to combine these rules using simple OR or AND logic (respectively rule_list and mandatory_rule_list).
+You need to set SMTP_SECURITY in the config file to one of the following options:
+ - 'tls'
+ - 'starttls'
+ - 'none'
+
+ Refer to your email provider to determine which you require.
+
+## PostProcess Modules
+Pastehunter comes with a couple of post process modules that extact useful data from pastes or pass them to other services
+The following are default modules:
+ - Base64 Decoders
+   - Cuckoo
+   - Viper
+
+## PreReqs
+
+### Pastebin
 
 You need a Pro account on pastebin that has access to the scraping API.
 https://pastebin.com/api_scraping_faq
 
-* Yara 
-* Python3
-* Elastic Search Kibana optional
+### GitHub
+Github needs an oauth token to stop it hitting the free ratelimit. 
+Create one at https://github.com/settings/tokens
+
+YOU DO NOT NEED TO GIVE IT ANY ACCESS PERMISSIONS
+
+# Installation
+
+## Local install 
+
+### Elastic Search
+https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html
+
+### Kibana
+https://www.elastic.co/guide/en/kibana/current/deb.html
 
-# Install.
+### Yara
+https://yara.readthedocs.io/en/latest/gettingstarted.html#compiling-and-installing-yara
+
+If you have yara errors check the installed version numbers for yara and yara-python match the lastest versions.
+
+### PasteHunter
+git clone https://github.com/kevthehermit/pastehunter
+
+### Python / Deps
+Python 3
+```pip3 install -r requirements.txt```
 
 ## Using Docker
 
@@ -36,47 +95,45 @@ Kibana is using the static IP address : 172.16.10.12 in the `esnet`  network
 Elasticsearch is running only on the localhost interface on default port 9200.
 The mount point is `/usr/share/elasticsearch/data` by default
 
+if elastic search fails to start and you see "max virtual memory areas vm.max_map_count [65530] likely too low"
+in the logs then try 
+
+`sudo sysctl -w vm.max_map_count=262144`
+
+https://elk-docker.readthedocs.io/#troubleshooting Paragraph starting As from version 5
+
 #### Pastehunter
 
 You can re-run the pastehunter script by doing `docker-compose up -d`
 Docker-compose will use already running instances of Elasticsearch and Kibana
 
-## Local install 
-
-### Elastic Search
-https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html
-
-### Kibana
-https://www.elastic.co/guide/en/kibana/current/deb.html
 
-### Yara
-https://yara.readthedocs.io/en/v3.6.0/gettingstarted.html#compiling-and-installing-yara
+# Configure
 
-Don't forget the python bindings
-```pip install yara-python```
+copy settings.json.sample to settings.json
+populate the details.
+For the scraping API you need to whitelist your IP on pastebin. No API key is required. See the link above
 
-If you have yara errors check the installed version numbers for yara and yara-python match the lastest versions.
+The logging level can be set to one of the following values. 
 
 
-### This little app
-git clone https://github.com/kevthehermit/pastehunter
+| Level    | Numerical |
+|----------|-----------|
+| CRITICAL | 50        |
+| ERROR    | 40        |
+| WARNING  | 30        |
+| INFO     | 20        |
+| DEBUG    | 10        |
+| NOTSET   | 0         |
 
-# Configure
+The default is INFO:20
 
-copy settings.conf.sample to settings.conf
-populate the details.
-For the scraping API you need to whitelist your IP on pastebin. No API key is required. See the link above
 
 # Running
 
-This needs python 3 as per the prereqs. 
-You can run it on its own with ```python3 pastehunter.py```
+Start the application with ```python3 pastehunter.py```
 
-Or you can set a cronjob to run this script every two minutes with a pastelimit of 200
+It may be useful to run in a screen to keep it running in the background. 
 
-```
-localadmin@pastebin:~/pastehunter$ cat /etc/cron.d/pastehunter
-# Run every 5 minutes
-*/2 * * * *   localadmin  cd /home/localadmin/pastehunter && python3 pastehunter.py >> /home/localadmin/pastehunter/cronlog.txt
-localadmin@pastebin:~/pastehunter$
-```
+## Service 
+Service config is coming 

YaraRules/api_keys.yar

@@ -50,3 +50,56 @@ rule google_api
     condition:
         all of them
 }
+
+rule slack_api
+{
+    meta:
+        author = "@ntddk"
+        info = "Part of PasteHunter"
+        reference = "https://github.com/kevthehermit/PasteHunter"
+
+    strings:
+        $a = /(xox(p|b|o|a)-[0-9]{9,12}-[0-9]{9,12}-[0-9]{9,12}-[a-z0-9]{32})/
+    condition:
+        all of them
+}
+
+rule github_api
+{
+    meta:
+        author = "@ntddk"
+        info = "Part of PasteHunter"
+        reference = "https://github.com/kevthehermit/PasteHunter"
+
+    strings:
+        $a = /[g|G][i|I][t|T][h|H][u|U][b|B].*[[\'|"]0-9a-zA-Z]{35,40}[\'|"]/
+    condition:
+        all of them
+}
+
+rule aws_api
+{
+    meta:
+        author = "@ntddk"
+        info = "Part of PasteHunter"
+        reference = "https://github.com/kevthehermit/PasteHunter"
+
+    strings:
+        $a = /AKIA[0-9A-Z]{16}/
+    condition:
+        all of them
+}
+
+rule heroku_api
+{
+    meta:
+        author = "@ntddk"
+        info = "Part of PasteHunter"
+        reference = "https://github.com/kevthehermit/PasteHunter"
+
+    strings:
+        $a = /[h|H][e|E][r|R][o|O][k|K][u|U].*[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}/
+    condition:
+        all of them
+}
+

YaraRules/base64.yar

@@ -10,9 +10,10 @@ rule b64_exe
         reference = "https://github.com/kevthehermit/PasteHunter"
 
     strings:
-        $b64_exe = /TV(oA|pB|pQ|qA|qQ|ro)/
+        $b64_exe = /\bTV(oA|pB|pQ|qA|qQ|ro)/
+        // Double b64 = VFZxUU
     condition:
-        $b64_exe
+        $b64_exe at 0
 
 }
 
@@ -88,4 +89,69 @@ rule b64_url
     condition:
         any of them
 
+}
+
+rule b64_doc
+{
+    meta:
+        author = "@KevTheHermit"
+        info = "Part of PasteHunter"
+        reference = "https://github.com/kevthehermit/PasteHunter"
+
+    strings:
+        $b64_doc = "0M8R4" // d0cf11
+    condition:
+        $b64_doc at 0
+
+}
+
+rule b64_rtf
+{
+    meta:
+        author = "@KevTheHermit"
+        info = "Part of PasteHunter"
+        reference = "https://github.com/kevthehermit/PasteHunter"
+
+    strings:
+        $b64_rtf = "e1xydGY" // {\rtf
+    condition:
+        $b64_rtf at 0
+
+}
+
+rule b64_docx
+{
+    meta:
+        author = "@KevTheHermit"
+        info = "Part of PasteHunter"
+        reference = "https://github.com/kevthehermit/PasteHunter"
+
+    strings:
+        $b64_zip = "UEs"
+        $docx1 = "d29yZC9fcmVsc" // word/_rel
+        $docx2 = "Zm9udFRhYmxl" // fontTable
+        $docx3 = "ZG9jUHJvcHM" // docProps
+        $docx4 = "Q29udGVudF9UeXBlcw" // Content_Types
+        $docx5 = "c2V0dGluZ3M" //settings
+    condition:
+        $b64_zip at 0 and 3 of ($docx*)
+
+}
+
+rule b64_xml_doc
+{
+    meta:
+        author = "@KevTheHermit"
+        info = "Part of PasteHunter"
+        reference = "https://github.com/kevthehermit/PasteHunter"
+
+    strings:
+        $b64_xml = "PD94bWwg"
+        $docx1 = "b3BlbmRvY3VtZW50" // opendocument
+        $docx2 = "InBhcmFncmFwaCI" // "paragraph"
+        $docx3 = "b2ZmaWNlL3dvcmQv" // office/word/
+        $docx4 = "RG9jdW1lbnRQcm9wZXJ0aWVz" // DocumentProperties
+    condition:
+        $b64_xml at 0 and 3 of ($docx*)
+
 }

YaraRules/blacklist.yar

@@ -0,0 +1,15 @@
+rule blacklist
+{
+    meta:
+        author = "@KevTheHermit"
+        info = "Part of PasteHunter"
+        reference = "https://github.com/kevthehermit/PasteHunter"
+
+    strings:
+        $a = "#EXTINF:" nocase // IPTV stream Lists.
+        $b = "--app-name=LeagueClient" nocase // League of Legends Debug Log
+        $c = "common.application_name: LeagueClient" // League of Legends Debug Log
+    condition:
+        any of them
+
+}

YaraRules/core_keywords.yar

@@ -15,11 +15,32 @@ rule core_keywords
         $enabled_sec = "enable secret" wide ascii nocase
         $enable_pass = "enable password" wide ascii nocase
         $ssh_priv = "BEGIN RSA PRIVATE KEY" wide ascii nocase
+        $openssh_priv = "BEGIN OPENSSH PRIVATE KEY" wide ascii nocase
+        $dsa_priv = "BEGIN DSA PRIVATE KEY" wide ascii nocase
+        $ec_priv = "BEGIN EC PRIVATE KEY" wide ascii nocase
         $pgp_priv = "BEGIN PGP PRIVATE KEY" wide ascii nocase
-        $DOX = " DOX" wide ascii nocase
         $hacked = "hacked by" wide ascii nocase
         $onion_url = /.*.\.onion/
     condition:
         any of them
 
-}
+}
+
+rule dox
+{
+    meta:
+        author = "@KevTheHermit"
+        info = "Part of PasteHunter"
+        reference = "https://github.com/kevthehermit/PasteHunter"
+
+    strings:
+        $dox = "DOX" wide ascii nocase fullword
+        $keyword1 = "name" wide ascii nocase
+        $keyword2 = "dob" wide ascii nocase
+        $keyword3 = "age" wide ascii nocase
+        $keyword4 = "password" wide ascii nocase
+        $keyword5 = "email" wide ascii nocase
+    condition:
+        $dox and 3 of ($keyword*)
+
+}