Name: Data Science Canberra Meeting
Start: 2017-07-24T16:00:00+10:00
End: 2017-07-24T19:00:00+10:00
Location: PwC Canberra

Exploiting Redundancy, Recurrency and Parallelism: How to Link Millions of Names and Addresses with Ten Lines of Code in Ten Minutes

Accurate and efficient record linkage is an open challenge of particular relevance to Australian Government Agencies, who recognise that so-called wicked social problems are best tackled by forming partnerships founded on large-scale data fusion. Names and addresses are the most common attributes on which data from different government agencies can be linked. We present a new method to link names and addresses between two large datasets, both of which have significant data quality issues. The most common approach for dealing with quality issues is to standardise raw data prior to linking. If a mistake is made in standardisation, however, it is usually impossible to recover from it to perform linkage correctly. By contrast, we show how data linkage can be implemented without standardisation. Empirical results show that approximately 91% of the links created between two large datasets from two government agencies were correct, despite significant data quality issues. Besides being accurate, our method is also highly efficient. Linking two datasets each containing millions of rows takes less than ten minutes. Our method is also easy to maintain. It can be implemented with ten SQL statements.

Dr Yuhang Zhang - AUSTRAC

Alex A.

Canberra R Users Group

Technology

Open Source

Big Data

Data Analytics

R Project for Statistical Computing

Data Visualization

Data Science Canberra Meeting

PwC Canberra

Share

Canberra R Users Group

Data Science Canberra Meeting

Canberra R Users Group

Details

Members are also interested in