- Topic: Hunting PI in the data haystack-A Streamlit in Snowflake detective story
- Date: Thursday 18th September at 6.00 pm - 8.00 pm
- Networking: 6 pm - 6.30 pm and 7.20 - 8 pm
- Presentation Time: 6.30 pm - 7.20 pm
- (includes welcome, presentation 30 mins & community marketplace)
- Location: Thoughtworks, Level 35/360 Collins St, Melbourne VIC 3000
- Ticket: Free of cost, however, registrations and RSVP are required!
- Sponsor: ThoughtWorks
Mangesh Jathar and Nigel Quinlan from MYOB will share their journey tackling the challenge of finding and classifying Personally Identifiable Information (PI) across a vast and continually growing data landscape - comprising over 94Tb of data in 50+ diverse domains.
Security and compliance play key roles in safeguarding sensitive customer information and there is increasing scrutiny brought on by recent incidents in the industry. To address the risk of unprotected or unnoticed PI, MYOB initiated a program to detect, classify, and monitor PI using a combination of Snowflake’s native capabilities and custom-built solutions. The team implemented a hybrid process featuring:
- Automated tagging of data columns for PI, leveraging both Snowflake’s system tags and MYOB-specific custom classifiers.
- REGEX-driven custom tagging, allowing adaptation to organisation-specific data patterns and requirements.
- Scheduled scanning tasks for each domain, integrating findings into a centralised audit framework and enabling incremental re-scans based on schema changes to minimise overhead.
- A Streamlit in Snowflake application to serve up records to domain owners and members
About Speakers
Mangesh
Mangesh Jathar is a Senior Data Engineer in MYOB's Data and AI group, working within the Data and Analytics Platform team in Melbourne.
With a deep experience in data engineering, dimensional modelling, and cloud warehousing, he specialises in developing and implementing scalable data solutions that empower MYOB's data-driven decision-making.
Mangesh's expertise includes designing high-performance data pipelines, optimising platform reliability, and fostering best practices n innovation in modern data engineering.
Nigel
Nigel Quinlan is a Data Engineer at MYOB, working within the Data and Analytics Platform team based in Melbourne (Cremorne). Nigel's work centres on collaboration, partnering closely with colleagues to support platform users and ensure their needs shape the evolution of MYOB's data infrastructure. Most at home exploring the data platform and working in Python, Nigel brings a people-first approach to developing practical solutions and fostering teamwork.
