An Empirical Study on Recommendation with Multiple Types of Feedback
Liang Tang*, LinkedIn Corp.; Bo Long, LinkedIn; Bee-Chung Chen, LinkedIn; Deepak Agarwal, LinkedIn
User feedback like clicks and ratings on recommended items pro-vides important information for recommender systems to predict users’ interests in unseen items. Most systems rely on models trained using a single type of feedback, e.g., ratings for movie recommendation and clicks for online news recommendation. How-ever, in addition to the primary feedback, many systems also allow users to provide other types of feedback, e.g., liking or sharing an article, or hiding all articles from a source. These additional feed-back potentially provides extra information for the recommendation models. To optimize user experience and business objectives,it is important for a recommender system to use both the primary feedback and additional feedback. This paper presents an empirical study on various training methods for incorporating multiple user feedback types based on LinkedIn recommendation products. We study three important problems that we face at LinkedIn: (1) Whether to send an email based on clicks and complaints, (2) how to rank updates in LinkedIn feeds based on clicks and hides and (3) how jointly optimize for viral actions and clicks in LinkedIn feeds. Extensive ofﬂine experiments on historical data show the effectiveness of these methods in different situations. Online A/B testing results further demonstrate the impact of these methods on LinkedIn production systems.
Filed under: Recommender Systems