Meetup #10 - Pair programming a parser for server logs to visualise API usage

About the system -
For a Java Spring Boot application having various API end points, will write an end point to visualize API usage on a daily basis (to begin with).

Details

• It will parse the nginx logs which are like this -
  10.30.1.141 - - [16/Dec/2019:04:07:23 +0000] "GET /service/v0/some/endpoint HTTP/1.1" 200 28 "-" "Java/1.8.0_101" "54.169.193.185"
  10.30.1.141 - - [16/Dec/2019:04:13:14 +0000] "GET / HTTP/1.1" 200 104 "-" "-" "125.64.94.197"
  10.30.4.51 - - [16/Dec/2019:04:14:56 +0000] "GET /service/v0/another/endpoint HTTP/1.1" 200 28 "-" "Java/1.8.0_101" "54.169.193.185"

and output stats like this -
{
"duration_from": "2019-12-16 00:00:00",
"duration_to": "2019-12-17 00:00:00",
"/service/v0/some/endpoint": {
"200": 12,
"400": 1,
"500": 2
},
"/service/v0/another/endpoint": {
"200": 10,
"400": 2,
"500": 7
}
}

• total hits/requests to server on a given day or in a date time range.
• hits against a specific API, with response code wise breakdown (200, 500 response codes).

The log file has one line for every request, with the following information -

• server time, API end point, response time, server response code etc.

A Linux grep command easily gives the desired output. I would like to call some grep commands from Java application and then put things in a more presentable format.

Note:

• This is not a prepared demo, so don't expect everything ready. I will be coding live and would like to have some partners to discuss while coding - https://en.wikipedia.org/wiki/Pair_programming.
• Please join with your mic, else there is no point!
• I do not have any power backup and have only a wifi connection which is dependent on power!
• People staying near Balewadi, who would like to attend face to face, can come to my home. Message me for details. I can host only 2 guests. Rest can attend online, via hangout.

About Speaker -
Senior Software Engineer at Talentica Software who has gained a lot of experience lately, working for Payu on their APIs for Citrus Wallet. He prefers live discussions over books and is currently on a mission to take his demonstration skills to the next level.