class_docstr: Class for handling KGE models which follows the ranking based protocol.
  
  Example
  -------
  >>> # create model and compile using user defined optimizer settings and
  >>> # user defined settings of an existing loss
  >>> from ampligraph.datasets import load_fb15k_237
  >>> from ampligraph.latent_features import ScoringBasedEmbeddingModel
  >>> from ampligraph.latent_features.loss_functions import SelfAdversarialLoss
  >>> import tensorflow as tf
  >>> X = load_fb15k_237()
  >>> loss = SelfAdversarialLoss({'margin': 0.1, 'alpha': 5, 'reduction': 'sum'})
  >>> model = ScoringBasedEmbeddingModel(eta=5,
  >>>                                    k=300,
  >>>                                    scoring_type='ComplEx',
  >>>                                    seed=0)
  >>> model.compile(optimizer='adam', loss=loss)
  >>> model.fit(X['train'],
  >>>           batch_size=10000,
  >>>           epochs=5)
  Epoch 1/5
  29/29 [==============================] - 3s 87ms/step - loss: 13496.5752
  Epoch 2/5
  29/29 [==============================] - 1s 36ms/step - loss: 13488.8682
  Epoch 3/5
  29/29 [==============================] - 1s 35ms/step - loss: 13436.2725
  Epoch 4/5
  29/29 [==============================] - 1s 35ms/step - loss: 13259.0840
  Epoch 5/5
  29/29 [==============================] - 1s 34ms/step - loss: 12977.0117
class_docstr_tok: ["Class","for","handling","KGE","models","which","follows","the","ranking","based","protocol",".","Example","--","--","--","-",">",">",">","#","create","model","and","compile","using","user","defined","optimizer","settings","and",">",">",">","#","user","defined","settings","of","an","existing","loss",">",">",">","from","ampligraph.datasets","import","load_fb15k_237",">",">",">","from","ampligraph.latent_features","import","ScoringBasedEmbeddingModel",">",">",">","from","ampligraph.latent_features.loss_functions","import","SelfAdversarialLoss",">",">",">","import","tensorflow","as","tf",">",">",">","X","=","load_fb15k_237","(",")",">",">",">","loss","=","SelfAdversarialLoss","(","{","'margin","'",":","0.1",",","'alpha","'",":","5",",","'reduction","'",":","'sum","'","}",")",">",">",">","model","=","ScoringBasedEmbeddingModel","(","eta=5",",",">",">",">","k=300",",",">",">",">","scoring_type='ComplEx","'",",",">",">",">","seed=0",")",">",">",">","model.compile","(","optimizer='adam","'",",","loss=loss",")",">",">",">","model.fit","(","X","[","'train","'","]",",",">",">",">","batch_size=10000",",",">",">",">","epochs=5",")","Epoch","1\/5","29\/29","[","==============================","]","-","3s","87ms\/step","-","loss",":","13496.5752","Epoch","2\/5","29\/29","[","==============================","]","-","1s","36ms\/step","-","loss",":","13488.8682","Epoch","3\/5","29\/29","[","==============================","]","-","1s","35ms\/step","-","loss",":","13436.2725","Epoch","4\/5","29\/29","[","==============================","]","-","1s","35ms\/step","-","loss",":","13259.0840","Epoch","5\/5","29\/29","[","==============================","]","-","1s","34ms\/step","-","loss",":","12977.0117"]
docstr: Calibrate predictions.
  
  The method implements the heuristics described in :cite:`calibration`,
  using Platt scaling :cite:`platt1999probabilistic`.
  
  The calibrated predictions can be obtained with :meth:`predict_proba`
  after calibration is done.
  
  Ideally, calibration should be performed on a validation set that was not used to train the embeddings.
  
  There are two modes of operation, depending on the availability of negative triples:
  
  #. Both positive and negative triples are provided via ``X_pos`` and ``X_neg`` respectively.         The optimization is done using a second-order method (limited-memory BFGS),         therefore no hyperparameter needs to be specified.
  
  #. Only positive triples are provided, and the negative triples are generated by corruptions,         just like it is done in training or evaluation. The optimization is done using a first-order method (ADAM),         therefore ``batches_count`` and ``epochs`` must be specified.
  
  
  Calibration is highly dependent on the base rate of positive triples.
  Therefore, for mode (2) of operation, the user is required to provide the ``positive_base_rate`` argument.
  For mode (1), that can be inferred automatically by the relative sizes of the positive and negative sets,
  but the user can override this behaviour by providing a value to ``positive_base_rate``.
  
  Defining the positive base rate is the biggest challenge when calibrating without negatives. That depends on
  the user choice of triples to be evaluated during test time.
  Let's take the WN11 dataset as an example: it has around 50% positives triples on both the validation set
  and test set, so the positive base rate follows to be 50%. However, should the user resample it to have
  75% positives and 25% negatives, the previous calibration would be degraded. The user must recalibrate
  the model with a 75% positive base rate. Therefore, this parameter depends on how the user handles the
  dataset and cannot be determined automatically or a priori.
  
  .. Note ::
      :cite:`calibration` `calibration experiments available here
      <https://github.com/Accenture/AmpliGraph/tree/paper/ICLR-20/experiments/ICLR-20>`_.
  
  
  Parameters
  ----------
  X_pos : np.array, shape (n,3) or str or GraphDataLoader or AbstractGraphPartitioner
      Data OR Filename of the data file OR Data Handle to be used as positive triples.
  X_neg : np.array, shape (n,3) or str or GraphDataLoader or AbstractGraphPartitioner
      Data OR Filename of the data file OR Data Handle to be used as negative triples.
  
      If `None`, the negative triples are generated via corruptions
      and the user must provide a positive base rate instead.
  
  positive_base_rate: float
      Base rate of positive statements.
  
      For example, if we assume there is an even chance for any query to be true, the base rate would be 50%.
  
      If ``X_neg`` is provided and ``positive_base_rate=None``, the relative sizes of ``X_pos`` and ``X_neg``
      will be used to determine the base rate. Say we have 50 positive triples and 200 negative
      triples, the positive base rate will be assumed to be :math:`\frac{50}{(50+200)} = \frac{1}{5} = 0.2`.
  
      This value must be :math:`\in [0,1]`.
  batches_size: int
      Batch size for positives.
  epochs: int
      Number of epochs used to train the Platt scaling model.
      Only applies when ``X_neg=None``.
  verbose: bool
      Verbosity (default: `False`).
  
  Example
  -------
  >>> from ampligraph.datasets import load_fb15k_237
  >>> from ampligraph.latent_features import ScoringBasedEmbeddingModel
  >>> import numpy as np
  >>> dataset = load_fb15k_237()
  >>> model = ScoringBasedEmbeddingModel(eta=5,
  >>>                                    k=300,
  >>>                                    scoring_type='ComplEx')
  >>> model.compile(optimizer='adam', loss='nll')
  >>> model.fit(dataset['train'],
  >>>           batch_size=10000,
  >>>           epochs=5)
  >>> print('Raw scores (sorted):', np.sort(model.predict(dataset['test'])))
  >>> print('Indices obtained by sorting (scores):', np.argsort(model.predict(dataset['test'])))
  Raw scores (sorted): [-1.0689778   -0.42082012  -0.39887887 ...  3.261838  3.2755773  3.2768354 ]
  Indices obtained by sorting (scores): [ 3834 18634  4066 ...  6237 13633 10961]
  >>> model.calibrate(dataset['test'],
  >>>                 batch_size=10000,
  >>>                 positive_base_rate=0.9,
  >>>                 epochs=100)
  >>> print('Calibrated scores (sorted):', np.sort(model.predict_proba(dataset['test'])))
  >>> print('Indices obtained by sorting (Calibrated):', np.argsort(model.predict_proba(dataset['test'])))
  Calibrated scores (sorted): [0.49547982 0.5396996  0.54118955 ... 0.7624245  0.7631044  0.76316655]
  Indices obtained by sorting (Calibrated): [ 3834 18634  4066 ...  6237 13633 10961]
docstr_tok: ["Calibrate","predictions",".","The","method","implements","the","heuristics","described","in",":","cite",":","`","calibration","`",",","using","Platt","scaling",":","cite",":","`","platt1999probabilistic","`",".","The","calibrated","predictions","can","be","obtained","with",":","meth",":","`","predict_proba","`","after","calibration","is","done",".","Ideally",",","calibration","should","be","performed","on","a","validation","set","that","was","not","used","to","train","the","embeddings",".","There","are","two","modes","of","operation",",","depending","on","the","availability","of","negative","triples",":","#",".","Both","positive","and","negative","triples","are","provided","via","``","X_pos","``","and","``","X_neg","``","respectively",".","The","optimization","is","done","using","a","second-order","method","(","limited-memory","BFGS",")",",","therefore","no","hyperparameter","needs","to","be","specified",".","#",".","Only","positive","triples","are","provided",",","and","the","negative","triples","are","generated","by","corruptions",",","just","like","it","is","done","in","training","or","evaluation",".","The","optimization","is","done","using","a","first-order","method","(","ADAM",")",",","therefore","``","batches_count","``","and","``","epochs","``","must","be","specified",".","Calibration","is","highly","dependent","on","the","base","rate","of","positive","triples",".","Therefore",",","for","mode","(","2",")","of","operation",",","the","user","is","required","to","provide","the","``","positive_base_rate","``","argument",".","For","mode","(","1",")",",","that","can","be","inferred","automatically","by","the","relative","sizes","of","the","positive","and","negative","sets",",","but","the","user","can","override","this","behaviour","by","providing","a","value","to","``","positive_base_rate","``",".","Defining","the","positive","base","rate","is","the","biggest","challenge","when","calibrating","without","negatives",".","That","depends","on","the","user","choice","of","triples","to","be","evaluated","during","test","time",".","Let","'s","take","the","WN11","dataset","as","an","example",":","it","has","around","50","%","positives","triples","on","both","the","validation","set","and","test","set",",","so","the","positive","base","rate","follows","to","be","50","%",".","However",",","should","the","user","resample","it","to","have","75","%","positives","and","25","%","negatives",",","the","previous","calibration","would","be","degraded",".","The","user","must","recalibrate","the","model","with","a","75","%","positive","base","rate",".","Therefore",",","this","parameter","depends","on","how","the","user","handles","the","dataset","and","can","not","be","determined","automatically","or","a","priori",".","..","Note",":",":",":","cite",":","`","calibration","`","`","calibration","experiments","available","here","<","https",":","\/\/github.com\/Accenture\/AmpliGraph\/tree\/paper\/ICLR-20\/experiments\/ICLR-20",">","`","_",".","Parameters","--","--","--","--","--","X_pos",":","np.array",",","shape","(","n,3",")","or","str","or","GraphDataLoader","or","AbstractGraphPartitioner","Data","OR","Filename","of","the","data","file","OR","Data","Handle","to","be","used","as","positive","triples",".","X_neg",":","np.array",",","shape","(","n,3",")","or","str","or","GraphDataLoader","or","AbstractGraphPartitioner","Data","OR","Filename","of","the","data","file","OR","Data","Handle","to","be","used","as","negative","triples",".","If","`","None","`",",","the","negative","triples","are","generated","via","corruptions","and","the","user","must","provide","a","positive","base","rate","instead",".","positive_base_rate",":","float","Base","rate","of","positive","statements",".","For","example",",","if","we","assume","there","is","an","even","chance","for","any","query","to","be","true",",","the","base","rate","would","be","50","%",".","If","``","X_neg","``","is","provided","and","``","positive_base_rate=None","``",",","the","relative","sizes","of","``","X_pos","``","and","``","X_neg","``","will","be","used","to","determine","the","base","rate",".","Say","we","have","50","positive","triples","and","200","negative","triples",",","the","positive","base","rate","will","be","assumed","to","be",":","math",":","`","\\frac","{","50","}","{","(","50+200",")","}","=","\\frac","{","1","}","{","5","}","=","0.2","`",".","This","value","must","be",":","math",":","`","\\in","[","0,1","]","`",".","batches_size",":","int","Batch","size","for","positives",".","epochs",":","int","Number","of","epochs","used","to","train","the","Platt","scaling","model",".","Only","applies","when","``","X_neg=None","``",".","verbose",":","bool","Verbosity","(","default",":","`","False","`",")",".","Example","--","--","--","-",">",">",">","from","ampligraph.datasets","import","load_fb15k_237",">",">",">","from","ampligraph.latent_features","import","ScoringBasedEmbeddingModel",">",">",">","import","numpy","as","np",">",">",">","dataset","=","load_fb15k_237","(",")",">",">",">","model","=","ScoringBasedEmbeddingModel","(","eta=5",",",">",">",">","k=300",",",">",">",">","scoring_type='ComplEx","'",")",">",">",">","model.compile","(","optimizer='adam","'",",","loss='nll","'",")",">",">",">","model.fit","(","dataset","[","'train","'","]",",",">",">",">","batch_size=10000",",",">",">",">","epochs=5",")",">",">",">","print","(","'Raw","scores","(","sorted",")",":","'",",","np.sort","(","model.predict","(","dataset","[","'test","'","]",")",")",")",">",">",">","print","(","'Indices","obtained","by","sorting","(","scores",")",":","'",",","np.argsort","(","model.predict","(","dataset","[","'test","'","]",")",")",")","Raw","scores","(","sorted",")",":","[","-1.0689778","-0.42082012","-0.39887887","...","3.261838","3.2755773","3.2768354","]","Indices","obtained","by","sorting","(","scores",")",":","[","3834","18634","4066","...","6237","13633","10961","]",">",">",">","model.calibrate","(","dataset","[","'test","'","]",",",">",">",">","batch_size=10000",",",">",">",">","positive_base_rate=0.9",",",">",">",">","epochs=100",")",">",">",">","print","(","'Calibrated","scores","(","sorted",")",":","'",",","np.sort","(","model.predict_proba","(","dataset","[","'test","'","]",")",")",")",">",">",">","print","(","'Indices","obtained","by","sorting","(","Calibrated",")",":","'",",","np.argsort","(","model.predict_proba","(","dataset","[","'test","'","]",")",")",")","Calibrated","scores","(","sorted",")",":","[","0.49547982","0.5396996","0.54118955","...","0.7624245","0.7631044","0.76316655","]","Indices","obtained","by","sorting","(","Calibrated",")",":","[","3834","18634","4066","...","6237","13633","10961","]"]
code: def calibrate(
      self,
      X_pos,
      X_neg=None,
      positive_base_rate=None,
      batch_size=32,
      epochs=50,
      verbose=0,
  ):
      """Calibrate predictions.
  
          The method implements the heuristics described in :cite:`calibration`,
          using Platt scaling :cite:`platt1999probabilistic`.
  
          The calibrated predictions can be obtained with :meth:`predict_proba`
          after calibration is done.
  
          Ideally, calibration should be performed on a validation set that was not used to train the embeddings.
  
          There are two modes of operation, depending on the availability of negative triples:
  
          #. Both positive and negative triples are provided via ``X_pos`` and ``X_neg`` respectively. \
          The optimization is done using a second-order method (limited-memory BFGS), \
          therefore no hyperparameter needs to be specified.
  
          #. Only positive triples are provided, and the negative triples are generated by corruptions, \
          just like it is done in training or evaluation. The optimization is done using a first-order method (ADAM), \
          therefore ``batches_count`` and ``epochs`` must be specified.
  
  
          Calibration is highly dependent on the base rate of positive triples.
          Therefore, for mode (2) of operation, the user is required to provide the ``positive_base_rate`` argument.
          For mode (1), that can be inferred automatically by the relative sizes of the positive and negative sets,
          but the user can override this behaviour by providing a value to ``positive_base_rate``.
  
          Defining the positive base rate is the biggest challenge when calibrating without negatives. That depends on
          the user choice of triples to be evaluated during test time.
          Let's take the WN11 dataset as an example: it has around 50% positives triples on both the validation set
          and test set, so the positive base rate follows to be 50%. However, should the user resample it to have
          75% positives and 25% negatives, the previous calibration would be degraded. The user must recalibrate
          the model with a 75% positive base rate. Therefore, this parameter depends on how the user handles the
          dataset and cannot be determined automatically or a priori.
  
          .. Note ::
              :cite:`calibration` `calibration experiments available here
              <https://github.com/Accenture/AmpliGraph/tree/paper/ICLR-20/experiments/ICLR-20>`_.
  
  
          Parameters
          ----------
          X_pos : np.array, shape (n,3) or str or GraphDataLoader or AbstractGraphPartitioner
              Data OR Filename of the data file OR Data Handle to be used as positive triples.
          X_neg : np.array, shape (n,3) or str or GraphDataLoader or AbstractGraphPartitioner
              Data OR Filename of the data file OR Data Handle to be used as negative triples.
  
              If `None`, the negative triples are generated via corruptions
              and the user must provide a positive base rate instead.
  
          positive_base_rate: float
              Base rate of positive statements.
  
              For example, if we assume there is an even chance for any query to be true, the base rate would be 50%.
  
              If ``X_neg`` is provided and ``positive_base_rate=None``, the relative sizes of ``X_pos`` and ``X_neg``
              will be used to determine the base rate. Say we have 50 positive triples and 200 negative
              triples, the positive base rate will be assumed to be :math:`\\frac{50}{(50+200)} = \\frac{1}{5} = 0.2`.
  
              This value must be :math:`\\in [0,1]`.
          batches_size: int
              Batch size for positives.
          epochs: int
              Number of epochs used to train the Platt scaling model.
              Only applies when ``X_neg=None``.
          verbose: bool
              Verbosity (default: `False`).
  
          Example
          -------
          >>> from ampligraph.datasets import load_fb15k_237
          >>> from ampligraph.latent_features import ScoringBasedEmbeddingModel
          >>> import numpy as np
          >>> dataset = load_fb15k_237()
          >>> model = ScoringBasedEmbeddingModel(eta=5,
          >>>                                    k=300,
          >>>                                    scoring_type='ComplEx')
          >>> model.compile(optimizer='adam', loss='nll')
          >>> model.fit(dataset['train'],
          >>>           batch_size=10000,
          >>>           epochs=5)
          >>> print('Raw scores (sorted):', np.sort(model.predict(dataset['test'])))
          >>> print('Indices obtained by sorting (scores):', np.argsort(model.predict(dataset['test'])))
          Raw scores (sorted): [-1.0689778   -0.42082012  -0.39887887 ...  3.261838  3.2755773  3.2768354 ]
          Indices obtained by sorting (scores): [ 3834 18634  4066 ...  6237 13633 10961]
          >>> model.calibrate(dataset['test'],
          >>>                 batch_size=10000,
          >>>                 positive_base_rate=0.9,
          >>>                 epochs=100)
          >>> print('Calibrated scores (sorted):', np.sort(model.predict_proba(dataset['test'])))
          >>> print('Indices obtained by sorting (Calibrated):', np.argsort(model.predict_proba(dataset['test'])))
          Calibrated scores (sorted): [0.49547982 0.5396996  0.54118955 ... 0.7624245  0.7631044  0.76316655]
          Indices obtained by sorting (Calibrated): [ 3834 18634  4066 ...  6237 13633 10961]
  
          """
      self.is_calibrated = False
      data_handler_calibrate_pos = data_adapter.DataHandler(
          X_pos,
          batch_size=batch_size,
          dataset_type="test",
          epochs=epochs,
          use_filter=False,
          use_indexer=self.data_indexer,
      )
  
      pos_size = (
          data_handler_calibrate_pos._parent_adapter.get_data_size()
      )
      neg_size = pos_size
  
      if X_neg is None:
          assert (
              positive_base_rate is not None
          ), "Please provide the negatives or positive base rate!"
          self.is_calibrate_with_corruption = True
      else:
          self.is_calibrate_with_corruption = False
  
          pos_batch_count = int(np.ceil(pos_size / batch_size))
  
          data_handler_calibrate_neg = data_adapter.DataHandler(
              X_neg,
              batch_size=batch_size,
              dataset_type="test",
              epochs=epochs,
              use_filter=False,
              use_indexer=self.data_indexer,
          )
  
          neg_size = (
              data_handler_calibrate_neg._parent_adapter.get_data_size()
          )
          neg_batch_count = int(np.ceil(neg_size / batch_size))
  
          if pos_batch_count != neg_batch_count:
              batch_size_neg = int(np.ceil(neg_size / pos_batch_count))
              data_handler_calibrate_neg = data_adapter.DataHandler(
                  X_neg,
                  batch_size=batch_size_neg,
                  dataset_type="test",
                  epochs=epochs,
                  use_filter=False,
                  use_indexer=self.data_indexer,
              )
  
          if positive_base_rate is None:
              positive_base_rate = pos_size / (pos_size + neg_size)
  
      if positive_base_rate is not None and (
          positive_base_rate <= 0 or positive_base_rate >= 1
      ):
          raise ValueError(
              "positive_base_rate must be a value between 0 and 1."
          )
  
      self.calibration_layer = CalibrationLayer(
          pos_size, neg_size, positive_base_rate
      )
      calibrate_function = self.make_calibrate_function()
  
      optimizer = tf.keras.optimizers.Adam()
  
      if not self.is_calibrate_with_corruption:
          negative_iterator = iter(
              data_handler_calibrate_neg.enumerate_epochs()
          )
  
      for _, iterator in data_handler_calibrate_pos.enumerate_epochs(
          True
      ):
          if not self.is_calibrate_with_corruption:
              _, neg_handle = next(negative_iterator)
  
          with data_handler_calibrate_pos.catch_stop_iteration():
              for step in data_handler_calibrate_pos.steps():
                  if self.is_calibrate_with_corruption:
                      scores_pos, scores_neg = calibrate_function(
                          iterator
                      )
  
                  else:
                      scores_pos = calibrate_function(iterator)
                      with data_handler_calibrate_neg.catch_stop_iteration():
                          scores_neg = calibrate_function(neg_handle)
  
                  with tf.GradientTape() as tape:
                      out = self.calibration_layer(
                          scores_pos, scores_neg, 1
                      )
  
                  gradients = tape.gradient(
                      out, self.calibration_layer._trainable_weights
                  )
                  # update the trainable params
                  optimizer.apply_gradients(
                      zip(
                          gradients,
                          self.calibration_layer._trainable_weights,
                      )
                  )
      self.is_calibrated = True
  
imports: import tensorflow
  import copy
  import shelve
  import pickle
  import numpy
  import os
  import tempfile
  import logging
  from ampligraph.evaluation.metrics import mrr_score, hits_at_n_score, mr_score
  from ampligraph.datasets import data_adapter
  from ampligraph.datasets.partitioned_data_manager import PartitionDataManager
  from ampligraph.latent_features.layers.scoring.AbstractScoringLayer import SCORING_LAYER_REGISTRY
  from ampligraph.latent_features.layers.encoding import EmbeddingLookupLayer
  from ampligraph.latent_features.layers.calibration import CalibrationLayer
  from ampligraph.latent_features.layers.corruption_generation import CorruptionGenerationLayerTrain
  from ampligraph.datasets.data_indexer import DataIndexer
  from ampligraph.latent_features import optimizers
  from ampligraph.latent_features import loss_functions
  from ampligraph.evaluation import train_test_split_no_unseen
  from tensorflow.python.keras import callbacks
  from tensorflow.python.keras.engine import training_utils
  from tensorflow.python.eager import def_function
  from tensorflow.python.keras import metrics
  from tensorflow.python.keras.engine import compile_utils
label_desc: Use image node_id 36 for calling the ScoringBasedEmbeddingModel obj's underlying member method code with example usage: obj.calibrate(X_pos, X_neg, positive_base_rate, batch_size, epochs, verbose) without return types