class_docstr: Wraps a pretrained Huggingface tokenizer to work as an sklearn
  transformer
  
  From the `tokenizers docs
  <https://huggingface.co/docs/tokenizers/python/latest/index.html>`_:
  
  ::
  
      🤗 Tokenizers provides an implementation of today’s most used
      tokenizers, with a focus on performance and versatility.
  
  Use pretrained Hugging Face tokenizers in an sklearn compatible transformer.
  
  Examples
  --------
  >>> from skorch.hf import HuggingfacePretrainedTokenizer
  >>> # pass the model name to be downloaded
  >>> hf_tokenizer = HuggingfacePretrainedTokenizer('bert-base-uncased')
  >>> data = ['hello there', 'this is a text']
  >>> hf_tokenizer.fit(data)  # only loads the model
  >>> hf_tokenizer.transform(data)
  
  >>> # pass pretrained tokenizer as object
  >>> my_tokenizer = ...
  >>> hf_tokenizer = HuggingfacePretrainedTokenizer(my_tokenizer)
  >>> hf_tokenizer.fit(data)
  >>> hf_tokenizer.transform(data)
  
  >>> # use hyper params from pretrained tokenizer to fit on own data
  >>> hf_tokenizer = HuggingfacePretrainedTokenizer(
  ...     'bert-base-uncased', train=True, vocab_size=12345)
  >>> data = ...
  >>> hf_tokenizer.fit(data)  # fits new tokenizer on data
  >>> hf_tokenizer.transform(data)
  
  Parameters
  ----------
  tokenizer : str or os.PathLike or transformers.PreTrainedTokenizerFast
    If a string, the model id of a predefined tokenizer hosted inside a model
    repo on huggingface.co. Valid model ids can be located at the root-level,
    like bert-base-uncased, or namespaced under a user or organization name,
    like dbmdz/bert-base-german-cased. If a path, A path to a directory
    containing vocabulary files required by the tokenizer, e.g.,
    ./my_model_directory/. Else, should be an instantiated
    ``PreTrainedTokenizerFast``.
  
  train : bool (default=False)
    Whether to use the pre-trained tokenizer directly as is or to retrain it
    on your data. If you just want to use the pre-trained tokenizer without
    further modification, leave this parameter as False. However, if you want
    to fit the tokenizer on your own data (completely from scratch, forgetting
    what it has learned previously), set this argument to True. The latter
    option is useful if you want to use the same hyper-parameters as the
    pre-trained tokenizer but want the vocabulary to be fitted to your
    dataset. The vocabulary size of this new tokenizer can be set explicitly
    by passing the ``vocab_size`` argument.
  
  max_length : int (default=256)
    Maximum number of tokens used per sequence.
  
  return_tensors : one of None, str, 'pt', 'np', 'tf' (default='pt')
    What type of result values to return. By default, return a padded and
    truncated (to ``max_length``) PyTorch Tensor. Similarly, 'np' results in a
    padded and truncated numpy array. Tensorflow tensors are not supported
    officially supported but should also work. If None or str, return a list
    of lists instead. These lists are not padded or truncated, thus each row
    may have different numbers of elements.
  
  return_attention_mask : bool (default=True)
    Whether to return the attention mask.
  
  return_token_type_ids : bool (default=False)
    Whether to return the token type ids.
  
  return_length : bool (default=False)
    Whether to return the length of the encoded inputs.
  
  pad_token : str (default='[PAD]')
    A special token used to make arrays of tokens the same size for batching
    purpose. Will then be ignored by attention mechanisms.
  
  vocab_size : int or None (default=None)
    Change this parameter only if you use ``train=True``. In that case, this
    parameter will determine the vocabulary size of the newly trained
    tokenizer. If you set ``train=True`` but leave this parameter as None, the
    same vocabulary size as the one from the initial toknizer will be used.
  
  verbose : int (default=0)
    Whether the tokenizer should print more information and warnings.
  
  Attributes
  ----------
  vocabulary_ : dict
    A mapping of terms to feature indices.
  
  fast_tokenizer_ : transformers.PreTrainedTokenizerFast
    If you want to extract the Hugging Face tokenizer to use it without skorch,
    use this attribute.
  
  .. _tokenizers: https://huggingface.co/docs/tokenizers/python/latest/index.html
class_docstr_tok: ["Wraps","a","pretrained","Huggingface","tokenizer","to","work","as","an","sklearn","transformer","From","the","`","tokenizers","docs","<","https",":","\/\/huggingface.co\/docs\/tokenizers\/python\/latest\/index.html",">","`","_",":",":",":","\ud83e\udd17","Tokenizers","provides","an","implementation","of","today","\u2019","s","most","used","tokenizers",",","with","a","focus","on","performance","and","versatility",".","Use","pretrained","Hugging","Face","tokenizers","in","an","sklearn","compatible","transformer",".","Examples","--","--","--","--",">",">",">","from","skorch.hf","import","HuggingfacePretrainedTokenizer",">",">",">","#","pass","the","model","name","to","be","downloaded",">",">",">","hf_tokenizer","=","HuggingfacePretrainedTokenizer","(","'bert-base-uncased","'",")",">",">",">","data","=","[","'hello","there","'",",","'this","is","a","text","'","]",">",">",">","hf_tokenizer.fit","(","data",")","#","only","loads","the","model",">",">",">","hf_tokenizer.transform","(","data",")",">",">",">","#","pass","pretrained","tokenizer","as","object",">",">",">","my_tokenizer","=","...",">",">",">","hf_tokenizer","=","HuggingfacePretrainedTokenizer","(","my_tokenizer",")",">",">",">","hf_tokenizer.fit","(","data",")",">",">",">","hf_tokenizer.transform","(","data",")",">",">",">","#","use","hyper","params","from","pretrained","tokenizer","to","fit","on","own","data",">",">",">","hf_tokenizer","=","HuggingfacePretrainedTokenizer","(","...","'bert-base-uncased","'",",","train=True",",","vocab_size=12345",")",">",">",">","data","=","...",">",">",">","hf_tokenizer.fit","(","data",")","#","fits","new","tokenizer","on","data",">",">",">","hf_tokenizer.transform","(","data",")","Parameters","--","--","--","--","--","tokenizer",":","str","or","os.PathLike","or","transformers.PreTrainedTokenizerFast","If","a","string",",","the","model","id","of","a","predefined","tokenizer","hosted","inside","a","model","repo","on","huggingface.co",".","Valid","model","ids","can","be","located","at","the","root-level",",","like","bert-base-uncased",",","or","namespaced","under","a","user","or","organization","name",",","like","dbmdz\/bert-base-german-cased",".","If","a","path",",","A","path","to","a","directory","containing","vocabulary","files","required","by","the","tokenizer",",","e.g.",",",".\/my_model_directory\/",".","Else",",","should","be","an","instantiated","``","PreTrainedTokenizerFast","``",".","train",":","bool","(","default=False",")","Whether","to","use","the","pre-trained","tokenizer","directly","as","is","or","to","retrain","it","on","your","data",".","If","you","just","want","to","use","the","pre-trained","tokenizer","without","further","modification",",","leave","this","parameter","as","False",".","However",",","if","you","want","to","fit","the","tokenizer","on","your","own","data","(","completely","from","scratch",",","forgetting","what","it","has","learned","previously",")",",","set","this","argument","to","True",".","The","latter","option","is","useful","if","you","want","to","use","the","same","hyper-parameters","as","the","pre-trained","tokenizer","but","want","the","vocabulary","to","be","fitted","to","your","dataset",".","The","vocabulary","size","of","this","new","tokenizer","can","be","set","explicitly","by","passing","the","``","vocab_size","``","argument",".","max_length",":","int","(","default=256",")","Maximum","number","of","tokens","used","per","sequence",".","return_tensors",":","one","of","None",",","str",",","'pt","'",",","'np","'",",","'tf","'","(","default='pt","'",")","What","type","of","result","values","to","return",".","By","default",",","return","a","padded","and","truncated","(","to","``","max_length","``",")","PyTorch","Tensor",".","Similarly",",","'np","'","results","in","a","padded","and","truncated","numpy","array",".","Tensorflow","tensors","are","not","supported","officially","supported","but","should","also","work",".","If","None","or","str",",","return","a","list","of","lists","instead",".","These","lists","are","not","padded","or","truncated",",","thus","each","row","may","have","different","numbers","of","elements",".","return_attention_mask",":","bool","(","default=True",")","Whether","to","return","the","attention","mask",".","return_token_type_ids",":","bool","(","default=False",")","Whether","to","return","the","token","type","ids",".","return_length",":","bool","(","default=False",")","Whether","to","return","the","length","of","the","encoded","inputs",".","pad_token",":","str","(","default=","'","[","PAD","]","'",")","A","special","token","used","to","make","arrays","of","tokens","the","same","size","for","batching","purpose",".","Will","then","be","ignored","by","attention","mechanisms",".","vocab_size",":","int","or","None","(","default=None",")","Change","this","parameter","only","if","you","use","``","train=True","``",".","In","that","case",",","this","parameter","will","determine","the","vocabulary","size","of","the","newly","trained","tokenizer",".","If","you","set","``","train=True","``","but","leave","this","parameter","as","None",",","the","same","vocabulary","size","as","the","one","from","the","initial","toknizer","will","be","used",".","verbose",":","int","(","default=0",")","Whether","the","tokenizer","should","print","more","information","and","warnings",".","Attributes","--","--","--","--","--","vocabulary_",":","dict","A","mapping","of","terms","to","feature","indices",".","fast_tokenizer_",":","transformers.PreTrainedTokenizerFast","If","you","want","to","extract","the","Hugging","Face","tokenizer","to","use","it","without","skorch",",","use","this","attribute",".","..","_tokenizers",":","https",":","\/\/huggingface.co\/docs\/tokenizers\/python\/latest\/index.html"]
docstr: Load the pretrained tokenizer
  
  Parameters
  ----------
  X : iterable of str
    This parameter is ignored.
  
  y : None
    This parameter is ignored.
  
  fit_params : dict
    This parameter is ignored.
  
  Returns
  -------
  self : HuggingfacePretrainedTokenizer
    The fitted instance of the tokenizer.
docstr_tok: ["Load","the","pretrained","tokenizer","Parameters","--","--","--","--","--","X",":","iterable","of","str","This","parameter","is","ignored",".","y",":","None","This","parameter","is","ignored",".","fit_params",":","dict","This","parameter","is","ignored",".","Returns","--","--","--","-","self",":","HuggingfacePretrainedTokenizer","The","fitted","instance","of","the","tokenizer","."]
code: def fit(self, X, y=None, **fit_params):
      """Load the pretrained tokenizer
  
      Parameters
      ----------
      X : iterable of str
        This parameter is ignored.
  
      y : None
        This parameter is ignored.
  
      fit_params : dict
        This parameter is ignored.
  
      Returns
      -------
      self : HuggingfacePretrainedTokenizer
        The fitted instance of the tokenizer.
  
      """
      from transformers import AutoTokenizer
  
      # from sklearn, triggers a parameter validation
      # even though X is not used, we leave this check in for consistency
      if isinstance(X, str):
          raise ValueError(
              "Iterable over raw text documents expected, string object received."
          )
  
      if not self.train and (self.vocab_size is not None):
          raise ValueError(
              "Setting vocab_size has no effect if train=False"
          )
  
      if isinstance(self.tokenizer, (str, os.PathLike)):
          self.fast_tokenizer_ = AutoTokenizer.from_pretrained(
              self.tokenizer
          )
      else:
          self.fast_tokenizer_ = self.tokenizer
  
      if not self.train:
          self.fixed_vocabulary_ = True
      else:
          X = list(X)  # transformers tokenizer does not accept arrays
          vocab_size = (
              self.fast_tokenizer_.vocab_size
              if self.vocab_size is None
              else self.vocab_size
          )
          self.fast_tokenizer_ = (
              self.fast_tokenizer_.train_new_from_iterator(
                  X, vocab_size=vocab_size
              )
          )
          self.fixed_vocabulary_ = False
  
      return self
  
imports: import io
  import os
  import pathlib
  from copy import deepcopy
  from operator import itemgetter
  import numpy
  import torch
  from sklearn.base import BaseEstimator, TransformerMixin
  from skorch.callbacks import LRScheduler
  from skorch.dataset import unpack_data
  from skorch.utils import check_is_fitted, params_for
label_desc: Use image node_id 2 for calling the HuggingfacePretrainedTokenizer obj's underlying member method code with example usage: obj.fit(X, y) and returns: self