class_docstr: Spark Evaluator for diversity, coverage, novelty, serendipity
class_docstr_tok: ["Spark","Evaluator","for","diversity",",","coverage",",","novelty",",","serendipity"]
docstr: Initializer.
  
  This is the Spark version of diversity metrics evaluator.
  The methods of this class calculate the following diversity metrics:
  
  * Coverage - it includes two metrics:
      1. catalog_coverage, which measures the proportion of items that get recommended from the item catalog;
      2. distributional_coverage, which measures how unequally different items are recommended in the
         recommendations to all users.
  * Novelty - A more novel item indicates it is less popular, i.e. it gets recommended less frequently.
  * Diversity - The dissimilarity of items being recommended.
  * Serendipity - The "unusualness" or "surprise" of recommendations to a user. When 'col_relevance' is used,
      it indicates how "pleasant surprise" of recommendations is to a user.
  
  The metric definitions/formulations are based on the following references with modification:
  
  :Citation:
  
      G. Shani and A. Gunawardana, Evaluating Recommendation Systems,
      Recommender Systems Handbook pp. 257-297, 2010.
  
      Y.C. Zhang, D.Ó. Séaghdha, D. Quercia and T. Jambor, Auralist: introducing
      serendipity into music recommendation, WSDM 2012
  
      P. Castells, S. Vargas, and J. Wang, Novelty and diversity metrics for recommender systems:
      choice, discovery and relevance, ECIR 2011
  
      Eugene Yan, Serendipity: Accuracy’s unpopular best friend in Recommender Systems,
      eugeneyan.com, April 2020
  
  Args:
      train_df (pyspark.sql.DataFrame): Data set with historical data for users and items they
          have interacted with; contains col_user, col_item. Assumed to not contain any duplicate rows.
          Interaction here follows the *item choice model* from Castells et al.
      reco_df (pyspark.sql.DataFrame): Recommender's prediction output, containing col_user, col_item,
          col_relevance (optional). Assumed to not contain any duplicate user-item pairs.
      item_feature_df (pyspark.sql.DataFrame): (Optional) It is required only when item_sim_measure='item_feature_vector'.
          It contains two columns: col_item and features (a feature vector).
      item_sim_measure (str): (Optional) This column indicates which item similarity measure to be used.
          Available measures include item_cooccurrence_count (default choice) and item_feature_vector.
      col_user (str): User id column name.
      col_item (str): Item id column name.
      col_relevance (str): Optional. This column indicates whether the recommended item is actually
          relevant to the user or not.
docstr_tok: ["Initializer",".","This","is","the","Spark","version","of","diversity","metrics","evaluator",".","The","methods","of","this","class","calculate","the","following","diversity","metrics",":","*","Coverage","-","it","includes","two","metrics",":","1.","catalog_coverage",",","which","measures","the","proportion","of","items","that","get","recommended","from","the","item","catalog",";","2.","distributional_coverage",",","which","measures","how","unequally","different","items","are","recommended","in","the","recommendations","to","all","users",".","*","Novelty","-","A","more","novel","item","indicates","it","is","less","popular",",","i.e",".","it","gets","recommended","less","frequently",".","*","Diversity","-","The","dissimilarity","of","items","being","recommended",".","*","Serendipity","-","The","``","unusualness","''","or","``","surprise","''","of","recommendations","to","a","user",".","When","'col_relevance","'","is","used",",","it","indicates","how","``","pleasant","surprise","''","of","recommendations","is","to","a","user",".","The","metric","definitions\/formulations","are","based","on","the","following","references","with","modification",":",":","Citation",":","G.","Shani","and","A.","Gunawardana",",","Evaluating","Recommendation","Systems",",","Recommender","Systems","Handbook","pp",".","257-297",",","2010",".","Y.C",".","Zhang",",","D.\u00d3",".","S\u00e9aghdha",",","D.","Quercia","and","T.","Jambor",",","Auralist",":","introducing","serendipity","into","music","recommendation",",","WSDM","2012","P.","Castells",",","S.","Vargas",",","and","J.","Wang",",","Novelty","and","diversity","metrics","for","recommender","systems",":","choice",",","discovery","and","relevance",",","ECIR","2011","Eugene","Yan",",","Serendipity",":","Accuracy","\u2019","s","unpopular","best","friend","in","Recommender","Systems",",","eugeneyan.com",",","April","2020","Args",":","train_df","(","pyspark.sql.DataFrame",")",":","Data","set","with","historical","data","for","users","and","items","they","have","interacted","with",";","contains","col_user",",","col_item",".","Assumed","to","not","contain","any","duplicate","rows",".","Interaction","here","follows","the","*","item","choice","model","*","from","Castells","et","al",".","reco_df","(","pyspark.sql.DataFrame",")",":","Recommender","'s","prediction","output",",","containing","col_user",",","col_item",",","col_relevance","(","optional",")",".","Assumed","to","not","contain","any","duplicate","user-item","pairs",".","item_feature_df","(","pyspark.sql.DataFrame",")",":","(","Optional",")","It","is","required","only","when","item_sim_measure='item_feature_vector","'",".","It","contains","two","columns",":","col_item","and","features","(","a","feature","vector",")",".","item_sim_measure","(","str",")",":","(","Optional",")","This","column","indicates","which","item","similarity","measure","to","be","used",".","Available","measures","include","item_cooccurrence_count","(","default","choice",")","and","item_feature_vector",".","col_user","(","str",")",":","User","id","column","name",".","col_item","(","str",")",":","Item","id","column","name",".","col_relevance","(","str",")",":","Optional",".","This","column","indicates","whether","the","recommended","item","is","actually","relevant","to","the","user","or","not","."]
code: def __init__(
      self,
      train_df,
      reco_df,
      item_feature_df=None,
      item_sim_measure=DEFAULT_ITEM_SIM_MEASURE,
      col_user=DEFAULT_USER_COL,
      col_item=DEFAULT_ITEM_COL,
      col_relevance=None,
  ):
      """Initializer.
  
      This is the Spark version of diversity metrics evaluator.
      The methods of this class calculate the following diversity metrics:
  
      * Coverage - it includes two metrics:
          1. catalog_coverage, which measures the proportion of items that get recommended from the item catalog;
          2. distributional_coverage, which measures how unequally different items are recommended in the
             recommendations to all users.
      * Novelty - A more novel item indicates it is less popular, i.e. it gets recommended less frequently.
      * Diversity - The dissimilarity of items being recommended.
      * Serendipity - The "unusualness" or "surprise" of recommendations to a user. When 'col_relevance' is used,
          it indicates how "pleasant surprise" of recommendations is to a user.
  
      The metric definitions/formulations are based on the following references with modification:
  
      :Citation:
  
          G. Shani and A. Gunawardana, Evaluating Recommendation Systems,
          Recommender Systems Handbook pp. 257-297, 2010.
  
          Y.C. Zhang, D.Ó. Séaghdha, D. Quercia and T. Jambor, Auralist: introducing
          serendipity into music recommendation, WSDM 2012
  
          P. Castells, S. Vargas, and J. Wang, Novelty and diversity metrics for recommender systems:
          choice, discovery and relevance, ECIR 2011
  
          Eugene Yan, Serendipity: Accuracy’s unpopular best friend in Recommender Systems,
          eugeneyan.com, April 2020
  
      Args:
          train_df (pyspark.sql.DataFrame): Data set with historical data for users and items they
              have interacted with; contains col_user, col_item. Assumed to not contain any duplicate rows.
              Interaction here follows the *item choice model* from Castells et al.
          reco_df (pyspark.sql.DataFrame): Recommender's prediction output, containing col_user, col_item,
              col_relevance (optional). Assumed to not contain any duplicate user-item pairs.
          item_feature_df (pyspark.sql.DataFrame): (Optional) It is required only when item_sim_measure='item_feature_vector'.
              It contains two columns: col_item and features (a feature vector).
          item_sim_measure (str): (Optional) This column indicates which item similarity measure to be used.
              Available measures include item_cooccurrence_count (default choice) and item_feature_vector.
          col_user (str): User id column name.
          col_item (str): Item id column name.
          col_relevance (str): Optional. This column indicates whether the recommended item is actually
              relevant to the user or not.
      """
  
      self.train_df = train_df.select(col_user, col_item)
      self.col_user = col_user
      self.col_item = col_item
      self.sim_col = DEFAULT_SIMILARITY_COL
      self.df_cosine_similarity = None
      self.df_user_item_serendipity = None
      self.df_user_serendipity = None
      self.avg_serendipity = None
      self.df_item_novelty = None
      self.avg_novelty = None
      self.df_intralist_similarity = None
      self.df_user_diversity = None
      self.avg_diversity = None
      self.item_feature_df = item_feature_df
      self.item_sim_measure = item_sim_measure
  
      if col_relevance is None:
          self.col_relevance = DEFAULT_RELEVANCE_COL
          # relevance term, default is 1 (relevant) for all
          self.reco_df = reco_df.select(
              col_user, col_item, F.lit(1.0).alias(self.col_relevance)
          )
      else:
          self.col_relevance = col_relevance
          self.reco_df = reco_df.select(
              col_user,
              col_item,
              F.col(self.col_relevance).cast(DoubleType()),
          )
  
      if self.item_sim_measure == "item_feature_vector":
          self.col_item_features = DEFAULT_ITEM_FEATURES_COL
          required_schema = StructType(
              (
                  StructField(self.col_item, IntegerType()),
                  StructField(self.col_item_features, VectorUDT()),
              )
          )
          if self.item_feature_df is not None:
              if str(required_schema) != str(item_feature_df.schema):
                  raise Exception(
                      "Incorrect schema! item_feature_df should have schema "
                      f"{str(required_schema)} but have {str(item_feature_df.schema)}"
                  )
          else:
              raise Exception(
                  "item_feature_df not specified! item_feature_df must be provided "
                  "if choosing to use item_feature_vector to calculate item similarity. "
                  f"item_feature_df should have schema {str(required_schema)}"
              )
  
      # check if reco_df contains any user_item pairs that are already shown in train_df
      count_intersection = (
          self.train_df.select(self.col_user, self.col_item)
          .intersect(self.reco_df.select(self.col_user, self.col_item))
          .count()
      )
  
      if count_intersection != 0:
          raise Exception(
              "reco_df should not contain any user_item pairs that are already shown in train_df"
          )
  
imports: import numpy
  from recommenders.utils.constants import DEFAULT_PREDICTION_COL, DEFAULT_USER_COL, DEFAULT_ITEM_COL, DEFAULT_RATING_COL, DEFAULT_RELEVANCE_COL, DEFAULT_SIMILARITY_COL, DEFAULT_ITEM_FEATURES_COL, DEFAULT_ITEM_SIM_MEASURE, DEFAULT_TIMESTAMP_COL, DEFAULT_K, DEFAULT_THRESHOLD
label_desc: Use image node_id 1 to create a new SparkDiversityEvaluation object  with example: obj = SparkDiversityEvaluation(train_df, reco_df, item_feature_df, item_sim_measure, col_user, col_item, col_relevance)