Data Cleaning publication categorizer

We propose a new data-driven transformation that facilitates many data mining, interpretation, and analysis tasks. Our approach, called MembershipMap, strives to extract the underlying sub-concepts of each raw attribute, and uses the orthogonal union of these sub-concepts to define a new space. The sub-concept soft labels of each point in the original space determine the position of that point in the new space.

The paper analyzes the problem of data cleansing and automatically identifying
potential errors in data sets. An overview of the diminutive amount of existing literature
concerning data cleansing is given. Methods for error detection that go beyond integrity
analysis are reviewed and presented. The applicable methods include: statistical outlier
detection, pattern matching, clustering, and data mining techniques. Some brief results
supporting the use of such methods are given. The future research directions necessary to
address the data cleansing problem are discussed.