Monitoring & Analysing Communication & Trade Events as Graphs

Flink is primarily known for its stream processing capabilities; nevertheless, it is also a powerful batch processing engine. When it comes to graph-analytics, Gelly offers a wide range of algorithms, implemented based on a diverse set of graph processing models. On the streaming side of things, when it comes to graphs one is not able to leverage graph algorithms directly; however, it is still possible to process real-time events in a way that allows for insightful graph visuals to be generated. This use-case comes from the finance domain, and concerns both a batch- and a stream- processing solution.

One of our clients wanted to solve two distinct problems:
a) analyse communication data and evaluate the effectiveness and efficiency of its current client-facing organisational structure, and;
b) process real-time trading events and understand their state and infer its obligations at any given time.

To address these problems, we combined 3 main technologies, Kafka, Flink and Elasticsearch, to build two distinct pipelines:

• an on-demand graph analytics pipeline, and;
• a stream processing pipeline for visualising graphs in real-time.

For the first pipeline, we relied on a two-phase architecture whereby in phase 1, e-mails were consumed from an event-log and were processed so that edges would be generated and stored in a queryable state (QS). In phase two, a user would query the QS and the returned output would be processed using Flink's Gelly library.

For the second pipeline, trade events were processed directly to extract graph edges that would be used to compose small-scale dependency graphs between selling and buying parties for a particular trading product.

The output of each of these pipelines was sunk into Elasticsearch, where a combination of visuals were used through insightful Kibana dashboards that helped increase the business's awareness, challenging organisation structures and aiding towards process optimisation.