class_docstr: Spark Evaluator for diversity, coverage, novelty, serendipity
class_docstr_tok: ["Spark","Evaluator","for","diversity",",","coverage",",","novelty",",","serendipity"]
docstr: Calculate novelty for each item. Novelty is computed as the minus logarithm of
  (number of interactions with item / total number of interactions). The definition of the metric
  is based on the following reference using the choice model (eqs. 1 and 6):
  
  :Citation:
  
      P. Castells, S. Vargas, and J. Wang, Novelty and diversity metrics for recommender systems:
      choice, discovery and relevance, ECIR 2011
  
  The novelty of an item can be defined relative to a set of observed events on the set of all items.
  These can be events of user choice (item "is picked" by a random user) or user discovery
  (item "is known" to a random user). The above definition of novelty reflects a factor of item popularity.
  High novelty values correspond to long-tail items in the density function, that few users have interacted
  with and low novelty values correspond to popular head items.
  
  Returns:
      pyspark.sql.dataframe.DataFrame: A dataframe with the following columns: col_item, item_novelty.
docstr_tok: ["Calculate","novelty","for","each","item",".","Novelty","is","computed","as","the","minus","logarithm","of","(","number","of","interactions","with","item","\/","total","number","of","interactions",")",".","The","definition","of","the","metric","is","based","on","the","following","reference","using","the","choice","model","(","eqs",".","1","and","6",")",":",":","Citation",":","P.","Castells",",","S.","Vargas",",","and","J.","Wang",",","Novelty","and","diversity","metrics","for","recommender","systems",":","choice",",","discovery","and","relevance",",","ECIR","2011","The","novelty","of","an","item","can","be","defined","relative","to","a","set","of","observed","events","on","the","set","of","all","items",".","These","can","be","events","of","user","choice","(","item","``","is","picked","''","by","a","random","user",")","or","user","discovery","(","item","``","is","known","''","to","a","random","user",")",".","The","above","definition","of","novelty","reflects","a","factor","of","item","popularity",".","High","novelty","values","correspond","to","long-tail","items","in","the","density","function",",","that","few","users","have","interacted","with","and","low","novelty","values","correspond","to","popular","head","items",".","Returns",":","pyspark.sql.dataframe.DataFrame",":","A","dataframe","with","the","following","columns",":","col_item",",","item_novelty","."]
code: def historical_item_novelty(self):
      """Calculate novelty for each item. Novelty is computed as the minus logarithm of
      (number of interactions with item / total number of interactions). The definition of the metric
      is based on the following reference using the choice model (eqs. 1 and 6):
  
      :Citation:
  
          P. Castells, S. Vargas, and J. Wang, Novelty and diversity metrics for recommender systems:
          choice, discovery and relevance, ECIR 2011
  
      The novelty of an item can be defined relative to a set of observed events on the set of all items.
      These can be events of user choice (item "is picked" by a random user) or user discovery
      (item "is known" to a random user). The above definition of novelty reflects a factor of item popularity.
      High novelty values correspond to long-tail items in the density function, that few users have interacted
      with and low novelty values correspond to popular head items.
  
      Returns:
          pyspark.sql.dataframe.DataFrame: A dataframe with the following columns: col_item, item_novelty.
      """
      if self.df_item_novelty is None:
          n_records = self.train_df.count()
          self.df_item_novelty = (
              self.train_df.groupBy(self.col_item)
              .count()
              .withColumn(
                  "item_novelty", -F.log2(F.col("count") / n_records)
              )
              .select(self.col_item, "item_novelty")
              .orderBy(self.col_item)
          )
      return self.df_item_novelty
  
imports: import numpy
  from recommenders.utils.constants import DEFAULT_PREDICTION_COL, DEFAULT_USER_COL, DEFAULT_ITEM_COL, DEFAULT_RATING_COL, DEFAULT_RELEVANCE_COL, DEFAULT_SIMILARITY_COL, DEFAULT_ITEM_FEATURES_COL, DEFAULT_ITEM_SIM_MEASURE, DEFAULT_TIMESTAMP_COL, DEFAULT_K, DEFAULT_THRESHOLD
label_desc: Use image node_id 10 for calling the SparkDiversityEvaluation obj's underlying member method code with example usage: obj.historical_item_novelty() and returns: self