class_docstr: Spark Evaluator for diversity, coverage, novelty, serendipity
class_docstr_tok: ["Spark","Evaluator","for","diversity",",","coverage",",","novelty",",","serendipity"]
docstr: Calculate serendipity of each item in the recommendations for each user.
  The metric definition is based on the following references:
  
  :Citation:
  
      Y.C. Zhang, D.Ó. Séaghdha, D. Quercia and T. Jambor, Auralist:
      introducing serendipity into music recommendation, WSDM 2012
  
      Eugene Yan, Serendipity: Accuracy’s unpopular best friend in Recommender Systems,
      eugeneyan.com, April 2020
  
  Returns:
      pyspark.sql.dataframe.DataFrame: A dataframe with columns: col_user, col_item, user_item_serendipity.
docstr_tok: ["Calculate","serendipity","of","each","item","in","the","recommendations","for","each","user",".","The","metric","definition","is","based","on","the","following","references",":",":","Citation",":","Y.C",".","Zhang",",","D.\u00d3",".","S\u00e9aghdha",",","D.","Quercia","and","T.","Jambor",",","Auralist",":","introducing","serendipity","into","music","recommendation",",","WSDM","2012","Eugene","Yan",",","Serendipity",":","Accuracy","\u2019","s","unpopular","best","friend","in","Recommender","Systems",",","eugeneyan.com",",","April","2020","Returns",":","pyspark.sql.dataframe.DataFrame",":","A","dataframe","with","columns",":","col_user",",","col_item",",","user_item_serendipity","."]
code: def user_item_serendipity(self):
      """Calculate serendipity of each item in the recommendations for each user.
      The metric definition is based on the following references:
  
      :Citation:
  
          Y.C. Zhang, D.Ó. Séaghdha, D. Quercia and T. Jambor, Auralist:
          introducing serendipity into music recommendation, WSDM 2012
  
          Eugene Yan, Serendipity: Accuracy’s unpopular best friend in Recommender Systems,
          eugeneyan.com, April 2020
  
      Returns:
          pyspark.sql.dataframe.DataFrame: A dataframe with columns: col_user, col_item, user_item_serendipity.
      """
      # for every col_user, col_item in reco_df, join all interacted items from train_df.
      # These interacted items are repeated for each item in reco_df for a specific user.
      if self.df_user_item_serendipity is None:
          self.df_cosine_similarity = self._get_cosine_similarity()
          self.df_user_item_serendipity = (
              self.reco_df.select(
                  self.col_user,
                  self.col_item,
                  F.col(self.col_item).alias(
                      "reco_item_tmp"
                  ),  # duplicate col_item to keep
              )
              .join(
                  self.train_df.select(
                      self.col_user,
                      F.col(self.col_item).alias("train_item_tmp"),
                  ),
                  on=[self.col_user],
              )
              .select(
                  self.col_user,
                  self.col_item,
                  F.least(
                      F.col("reco_item_tmp"), F.col("train_item_tmp")
                  ).alias("i1"),
                  F.greatest(
                      F.col("reco_item_tmp"), F.col("train_item_tmp")
                  ).alias("i2"),
              )
              .join(
                  self.df_cosine_similarity, on=["i1", "i2"], how="left"
              )
              .fillna(0)
              .groupBy(self.col_user, self.col_item)
              .agg(
                  F.mean(self.sim_col).alias(
                      "avg_item2interactedHistory_sim"
                  )
              )
              .join(self.reco_df, on=[self.col_user, self.col_item])
              .withColumn(
                  "user_item_serendipity",
                  (1 - F.col("avg_item2interactedHistory_sim"))
                  * F.col(self.col_relevance),
              )
              .select(
                  self.col_user, self.col_item, "user_item_serendipity"
              )
              .orderBy(self.col_user, self.col_item)
          )
      return self.df_user_item_serendipity
  
imports: import numpy
  from recommenders.utils.constants import DEFAULT_PREDICTION_COL, DEFAULT_USER_COL, DEFAULT_ITEM_COL, DEFAULT_RATING_COL, DEFAULT_RELEVANCE_COL, DEFAULT_SIMILARITY_COL, DEFAULT_ITEM_FEATURES_COL, DEFAULT_ITEM_SIM_MEASURE, DEFAULT_TIMESTAMP_COL, DEFAULT_K, DEFAULT_THRESHOLD
label_desc: Use image node_id 12 for calling the SparkDiversityEvaluation obj's underlying member method code with example usage: obj.user_item_serendipity() and returns: self