Building my first blue team tool: IOC Scraper + Aggregator
It's been awhile since I last posted anything to this blog, but I haven't forgotten about it. I live a busy life.. children, full time job, full time college student.. it can be overwhelming at times. Nevertheless, I return to share the first tool/script I've made and further document my never-ending journey into cybersecurity.
As I stated, I pushed my first actual script/tool to my Grimoire of Operations (my github repo). It is a modular python tool that extracts indicators of compromise (IOCs) from raw/unstructured text files. It's simply called ioc_scraper.
What it does?
The tool scrapes out commons IOCs like:
- IPv4 addresses
- URLs
- Domains
- Email Addresses
- SHA256 Hashes
- MD5 Hashes
You feed it a messy .txt file (a threat report, pastebin dump, or even a phishing email header) and it gives you structured output in the form of csv and json. It's a way to pull signal from noise and move faster when dealing with open-source intel or rough triage data.
Originally, I thought: "Wouldn't you already have clean IOCs if they were in a text file?" But that's not how this works in the real world. Threat intel often comes in scattered, unstructured formats. IOCs could be buried in paragraphs and you are stuck pulling the data you need manually.
What I learned?
More than anything, building this helped me understand python a bit deeper. I took a python scripting course for the summer semester, and while it has been a great way to force my hand with learning code.. it is not security focused. This has helped me better contextualize the ways I could/will be implementing python scripts into my workflows and understanding how python can serve to automate remedial tasks.
Future Ideas?
The script is modular, which means I can evolve it over time. Here's what next on my radar:
- Normalize obfuscated indicators
- Add VirusTotal or AbuselPDB enrichment
- Pull IOCs from PDFs or HTML reports
- Tag indicators by confidence level or source
- Correlate to MITRE ATT&CK Techniques
This is my first "spell" in my growing grimoire of operational tools I'm building for myself (and anyway who wants to use then too).
Repo
Check it out here: github.com/S0M3V3RM1N/grimoire-of-operations Path: /tools/ioc_scraper/
There's more to come. I'll post about it again when the next feature lands. For now, I'm just glad I took the time to slow down and build something that makes me better at what I do.
v3rm1n