September 2024 Reading Data & AI MeetUp (In-person Only)

Welcome to our September 2024 Data & AI Meetup. The first will be:

Creating cost transparency in multi-cloud enterprise: An OSS Approach

Gringotts is a cloud transparency tool that shows insights into a business enabling product teams spending in multi-cloud ecosystems, allowing them to drill down on current costs, get insights to make informed decisions about your products, and eventually control costs. It also supplies optimization recommendations to these teams to right-size on their cloud workloads to run on optimal costs & also gives them alerts on anomalies that occur on the cloud workloads in terms of daily expenditure. The solution we have built is completely built by using OSS frameworks, which allows it to be cloud agnostic in nature & effectively run at a very minimal operational expense. The solution itself consists of below mentioned components: - Data ingestion: This is done daily by used APIs from different cloud service providers which we consume using ETL services we have developed using .NET running on containerized workloads in Kubernetes. - Data transformation: This is again done used internal business metadata mapping which lets us associate cloud workloads (at a resource level) to business entities within the organization. This association creates the layer of association which helps us relate costs to a business entity & create transparency. - Data acceleration: Here we used an OSS tool called as Dremio (data lake engine) which helps us create a layer of abstraction on top of our data lake & serves the purpose of creating data dimensions & aggregations which can be used by different visualization tools. - Data visualization: Here again we have used an OSS tool called as Apache Superset (a visualization tool) to offer insights into the data that we have collected & transformed for our needs. Apache Superset offers immense capabilities like support for plugging into OAuth based RBAC, row-level security, custom roles, scheduled reports & alerts etc. Enabling it to be a perfect solution for our visualization needs. - Anomaly Detection & Optimization recommendations: Here we have used ML .NET to create a solution which offers anomaly detection to business entities. This is just a sneak peek into the solution, but I would love to talk more about it in depth in the meetup session on how we have solved this problem for our organization

Speaker: Nitin Sapru
I am a Senior Software Developer by day, aspiring tech superhero by night. I've been fighting the forces of software chaos in the IT industry for years, armed with my trusty C# and dotnet for better part of 9 years (and a secret stash of coffee).

There will be Pizza and Networking after the session.