Structuring Supervision trees and GenServers for efficiency and observability
Details
Talk Abstract:
Performance benefits don't Just Work because a project is in Elixir. You need to make sure to use OTP correctly to observe performance of each section of your application and tune performance.
We inherited a system that wasn't taking advantage of OTP's strengths. It was using external work queues and multiple OS processes and because of this, it wasn't any faster than implementations in other languages. By bringing the work queues into the BEAM, we were able to use async Tasks, but we still could improve the process by batching requests to the upstream API. With the work queues being in-process we also needed a restart recovery process. Combining these two features we created BufferedTask, which allows pre-populating the queue on start and maximizing the packing of batch requests while keeping the queues responsive by balancing batch size and wait time. This approach led to processing times dropping from 5-6 days to 5-6 hours.
With the success of BufferedTask on the smaller dataset we started to apply it to more sections of the domain and eventually larger data sets, but we ran into memory issues. BuferedTask delegates the actual running of batches to a Task.Supervisor, but we had been using the same Task.Supervisor for all BufferedTask, so we were unable to attribute memory usage to any one section of the domain. I'll show how we broke up the Task.Supervisor and restructured the Supervision tree so we could attribute memory and CPU usage to each BufferedTask and its batch Tasks.
This setup helped us find tricky bugs, but also allowed us to restrict memory usage when we wanted to target smaller hardware requirements. I'll show how using tools that ship with OTP allowed soft-memory limits and how we were able to repurpose the the restart recovery in BufferedTask to allow us to shed the work queue, but restore it when memory pressure lessened.
Speaker:
Luke Imhoff is the creator of the Elixir plugin for JetBrains IDEs and works at DockYard.
Location:
Go to Capital Factory's 1st floor front desk (the big room with the robot head across from the hotel bar) and ask for the Austin Elixir meetup room (usually Wall-E or Jonny-Five on the first floor) and they'll point you in the right direction. You might want to stop by the front desk to get to the parking validation ticket before the meetup too.
Parking:
Capital Factory has validated parking: when you go to the 1st or 16th floor ask the front desk for a parking voucher. The parking voucher will let you park for 50% off from 5pm-10pm (so ~$15 instead of ~$30) in the Omni Hotel garage under Capital Factory. To get into the garage turn north on Brazos from 7th Street. The garage entrance is the first downward ramp on the right at 733 Brazos (Street view https://www.google.com/maps/@30.2689199,-97.740976,3a,75y,88.67h,84.37t/data=!3m6!1e1!3m4!1s6eYOcptw_oovfL-UUMEM1Q!2e0!7i13312!8i6656!6m1!1e1 ) . There is cheaper lot parking at 406 E 7th Street (https://goo.gl/maps/LseRL3TGKC32)