Dedoop: efficient deduplication with Hadoop

Authors: 
Kolb, L; Thor, A; Rahm, E
Author: 
Kolb, L
Thor, A
Rahm, E

We demonstrate a powerful and easy-to-use tool called Dedoop (Deduplication with Hadoop) for MapReduce-based entity resolution (ER) of large datasets. Dedoop supports a browser-based specification of complex ER workflows including blocking and matching steps as well as the optional use of machine learning for the automatic generation of match classifiers. Specified workflows are automatically translated into MapReduce jobs for parallel execution on different Hadoop clusters. To achieve high performance Dedoop supports several advanced load balancing strategies.

Year: 
2012
Venue: 
Proceedings of the VLDB Endowment
URL: 
http://dl.acm.org/citation.cfm?id=2367527
Citations: 
0
Citations range: 
n/a