Fast Pattern Discovery in Big Data Mining Using Incremental Mining for Social Media Analysis
Nasser Al Riyami and Amal Al Hinai
Abstract
Social media websites generate high-volume, high-velocity data streams that are hard to deal with using traditional data-mining techniques. Most methods for mining frequent and sequential patterns of data currently in use require the whole data set to be reprocessed each time a new post is added to the data stream, which causes high computational costs, latency and poor scalability of the data stream with large data sets. The limitation is critical for real-time applications like trend detection, sentiment analysis, breaking news monitoring and targeted advertising, where the ability to capture emerging trends is vital in helping with timely decision-making and user engagement. A quick and incremental pattern-discovery framework for social media big data is proposed to overcome. The proposed work introduces an incremental mining algorithm that recovers known frequent and sequential patterns incrementally (i.e., incrementally enriching existing patterns as new data is added to the database and without rescanning the existing patterns). The framework's attributes of a compact data structure and an effective pruning mechanism consume lower memory space, help update patterns quickly, and allow continuous mining over the evolving social media streams, while retaining the quality and comprehensiveness of patterns. It has been evaluated with real Twitter/ Facebook posts from millions of users and has been measured for the time patterns that are discovered, memory requirements, pattern recall and latency to update the patterns. The experimental results show that the proposed method can save the mining time of 55%-65% and the memory usage 30%-40% compared with the non-incremental mining methods as a baseline, and can recall the patterns with a recall rate of 95%-97%. The major benefits are its early real-time trend detection, scale-up to big data, and reduced operational expenses, enabling it to be used in social media analytics, marketing intelligence or public sentiment monitoring systems.