Skip to content

Details

This is a hybrid event, and we'll share the Zoom link a couple of minutes before the meeting starts.

Topic:
Trust, but Verify: Testing Whether Your SIEM Actually Catches Attacks

Description:
Most security teams write detection rules, ship logs into their SIEM, and assume that if an attacker shows up, an alert will fire. Very few test that assumption, and fewer still re-test it after the next Sysmon update, agent upgrade, or rule change. Industry surveys keep finding the same thing: a large share of SIEM rules are broken or never fire, and nobody notices until an incident.
This talk walks through what I learned building SIEMbad, an open-source tool that runs Atomic Red Team attacks against a Wazuh-monitored Windows endpoint, then checks the SIEM for two separate things: did the telemetry arrive, and did an alert fire that can be traced back to the test? That distinction turns out to matter a lot. A missing alert might be a rule gap, a sensor gap, a SIEM outage, or a test that silently failed to run, and each one needs a different fix. We’ll look at the surprises along the way: why the Wazuh API can’t tell you whether a rule fired, how a seven-year-old Sysmon config quietly drops whole classes of events, why “the test executed” is never proof of detection, and how easy it is to overstate coverage.
We’ll also cover detection drift: snapshotting agent, Sysmon, audit policy, and rule state before every run so that when coverage drops, you can see what changed. The session ends with a 10–15 minute live demo of SIEMbad, from launching a pipeline to finding a gap, deploying a human-reviewed rule, and automatically retesting it, with the option to roll back.
Takeaways:

  • Separate “was it logged?” from “did it alert?” to tell sensor gaps from rule gaps
  • Recognize the common ways detections break silently, and how to catch drift between runs
  • Leave with a free, open-source workflow for continuously validating your own detections

Bio:
Vasily Rotin is a Director of IT at Kripalu Center for Yoga and Health with 10+ years of experience in IT and cybersecurity. He is completing his Master’s in Cybersecurity at Georgia Tech, where he built SIEMbad, an open-source tool for automated SIEM detection validation and gap analysis, as his practicum project. His interests include detection engineering, adversary emulation, and making security testing accessible.

Venue and Food:
This is a hybrid event!

Virtual: Zoom link will be provided at the time of the meeting.
Zoom link:

In-Person: We are meeting at 6:30 pm at Paragus IT at 112 Russell St, Hadley, MA. Please RSVP for the pizza and soda count.

Related topics

Events in Hadley, MA
Computer Security
Network Security
Hacking
Information Security
Computer Science

You may also like