Data Cleaning publication categorizer

Online market intelligence (OMI), in particular competitive
intelligence for product pricing, is a very important application
area for Web data extraction. However, OMI presents
non-trivial challenges to data extraction technology. Sophisticated
and highly parameterized navigation and extraction
tasks are required. On-the-fly data cleansing is necessary in
order two identify identical products from different suppliers.
It must be possible to smoothly define data flow scenarios
that merge and filter streams of extracted data stemming
from several Web sites and store the resulting data
into a data warehouse, where the data is subjected to market
intelligence analytics. Finally, the system must be highly
scalable, in order to be able to extract and process massive
amounts of data in a short time. Lixto (www.lixto.com),
a company offering data extraction tools and services, has
been providing OMI solutions for several customers. In this
paper we show how Lixto has tackled each of the above challenges
by improving and extending its original data extraction
software. Most importantly, we show how high scalability
is achieved through cloud computing. This paper also
features a case study from the computers and electronics
market.