The team at Energy Solutions spent a year building a robust data ingestion and query pipeline using OpenSearch to provide centralized data to a distributed suite of applications.
Along the way, they learned to question and rethink relational database assumptions and take fuzzy search customization and accuracy to the next level.
They implemented Pydantic wrappers around JSON responses to handle responses like native Python objects (along with other benefits we’ll discuss).
This solution addressed long-standing challenges, such as:
- Improving the performance of the per-row create/update/delete paradigm (in one case, leading to a ~9x faster data ingest + load!),
- Putting OpenSearch “aliases” to work to help track current vs archival data,
- Improving search relevancy,
- PII exposure reduction,
This presentation will walk through the decisions that led to moving to an OpenSearch-based solution that works within a traditional Django framework, how they tackled advanced topics like token analysis, and how they put OpenSearch aliases to work. We’ll also cover some of the cost-benefit equations, summarize the next phase of work in the project, and include real-time demonstrations of some concepts.
Schedule
5:45 pm - Doors Open / Social
6:30 pm - Meeting Start
7:30 pm - Wrap-up
8:00 pm - Prologue (Social in Scotts Addition)
All events are open to all skill levels. If you're just starting out, come to learn and feel free to ask questions. That's where the real learning happens. If you're a more seasoned developer, we'd love to learn from you. Despite what the Zen of Python says, there are usually more than one way to do something, so share your knowledge!
Join our Discord server! https://pyrva.org/discord
Want to sponsor PyRVA? https://pyrva.org/donate
Want to present? https://forms.gle/q8w3xziArWjud5f67
