Highly Scalable Recommendation System for Big Data Processing

Abstract

With the development of networks and IT technologies, users are searching for and purchasing the items they want from anywhere, regardless of location. Accordingly, various studies are being conducted on how to solve the scalability problem caused by the rapidly increasing data in the recommendation system. In this paper, we propose an item-based collaborative filtering method to which tag weight is applied and a recommendation method using the MapReduce method, a distributed parallel processing method. The proposed method classifies items by category in the preprocessing process and groups them according to the number of nodes for speed and efficiency. Data processing is performed through 4 MapReduce steps in each distributed node, and item tag weights are used in the similarity calculation to recommend better items to users. The top N items among the predicted values output through the last Reduce step are used for recommendation. Through experiments, it was confirmed that the proposed method efficiently processes a large amount of data, and the suitability of recommendation is improved compared to the existing item-based method.