class_docstr: Wraps a Hugging Face tokenizer to work as an sklearn transformer
  
  From the `tokenizers docs
  <https://huggingface.co/docs/tokenizers/python/latest/index.html>`_:
  
  ::
  
      🤗 Tokenizers provides an implementation of today’s most used
      tokenizers, with a focus on performance and versatility.
  
  Use of Hugging Face tokenizers for training on custom data using an sklearn
  compatible API.
  
  Examples
  --------
  >>> # train a BERT tokenizer from scratch
  >>> from tokenizers import Tokenizer
  >>> from tokenizers.models import WordPiece
  >>> from tokenizers import normalizers
  >>> from tokenizers.normalizers import Lowercase, NFD, StripAccents
  >>> from tokenizers.pre_tokenizers import Whitespace
  >>> from tokenizers.processors import TemplateProcessing
  >>> bert_tokenizer = Tokenizer(WordPiece(unk_token="[UNK]"))
  >>> normalizer = normalizers.Sequence([NFD(), Lowercase(), StripAccents()])
  >>> pre_tokenizer = Whitespace()
  >>> post_processor = TemplateProcessing(
  ...    single="[CLS] $A [SEP]",
  ...    pair="[CLS] $A [SEP] $B:1 [SEP]:1",
  ...    special_tokens=[
  ...        ("[CLS]", 1),
  ...        ("[SEP]", 2),
  ...    ],
  ... )
  >>> from skorch.hf import HuggingfaceTokenizer
  >>> hf_tokenizer = HuggingfaceTokenizer(
  ...     tokenizer=bert_tokenizer,
  ...     pre_tokenizer=pre_tokenizer,
  ...     post_processor=post_processor,
  ...     trainer__vocab_size=30522,
  ...     trainer__special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"],
  ... )
  >>> data = ['hello there', 'this is a text']
  >>> hf_tokenizer.fit(data)
  >>> hf_tokenizer.transform(data)
  
  In general, you can pass both initialized objects and uninitialized objects
  as parameters:
  
  .. code:: python
  
      # initialized
      HuggingfaceTokenizer(tokenizer=Tokenizer(model=WordPiece()))
      # uninitialized
      HuggingfaceTokenizer(tokenizer=Tokenizer, model=WordPiece)
  
  Both approaches work equally well and allow you to, for instance, grid
  search on the tokenizer parameters. However, it is recommended *not* to pass
  an initialized trainer. This is because the trainer will then be saved as an
  attribute on the object, which can be wasteful. Instead, it is best to leave
  the default ``trainer='auto'``, which results in the trainer being derived
  from the model.
  
  .. note::
  
      If you want to train the ``HuggingfaceTokenizer`` in parallel (e.g.
      during a grid search), you should probably set the environment variable
      ``TOKENIZERS_PARALLELISM=false``. Otherwise, you may experience slow
      downs or deadlocks.
  
  Parameters
  ----------
  tokenizer : tokenizers.Tokenizer
    The tokenizer to train.
  
  model : tokenizers.models.Model
    The model represents the actual tokenization algorithm, e.g. ``BPE``.
  
  trainer : tokenizers.trainers.Trainer or 'auto' (default='auto')
    Class responsible for training the tokenizer. If 'auto', the correct
    trainer will be inferred from the used model using
    ``model.get_trainer()``.
  
  normalizer : tokenizers.normalizers.Normalizer or None (default=None)
    Optional normalizer, e.g. for casting the text to lowercase.
  
  pre_tokenizer : tokenizers.pre_tokenizers.PreTokenizer or None (default=None)
    Optional pre-tokenization, e.g. splitting on space.
  
  post_processor : tokenizers.processors.PostProcessor
    Optional post-processor, mostly used to add special tokens for BERT etc.
  
  max_length : int (default=256)
    Maximum number of tokens used per sequence.
  
  return_tensors : one of None, str, 'pt', 'np', 'tf' (default='pt')
    What type of result values to return. By default, return a padded and
    truncated (to ``max_length``) PyTorch Tensor. Similarly, 'np' results in a
    padded and truncated numpy array. Tensorflow tensors are not supported
    officially supported but should also work. If None or str, return a list
    of lists instead. These lists are not padded or truncated, thus each row
    may have different numbers of elements.
  
  return_attention_mask : bool (default=True)
    Whether to return the attention mask.
  
  return_token_type_ids : bool (default=False)
    Whether to return the token type ids.
  
  return_length : bool (default=False)
    Whether to return the length of the encoded inputs.
  
  pad_token : str (default='[PAD]')
    A special token used to make arrays of tokens the same size for batching
    purpose. Will then be ignored by attention mechanisms.
  
  verbose : int (default=0)
    Whether the tokenizer should print more information and warnings.
  
  Attributes
  ----------
  vocabulary_ : dict
    A mapping of terms to feature indices.
  
  fast_tokenizer_ : transformers.PreTrainedTokenizerFast
    If you want to extract the Hugging Face tokenizer to use it without skorch,
    use this attribute.
  
  .. _tokenizers: https://huggingface.co/docs/tokenizers/python/latest/index.html
class_docstr_tok: ["Wraps","a","Hugging","Face","tokenizer","to","work","as","an","sklearn","transformer","From","the","`","tokenizers","docs","<","https",":","\/\/huggingface.co\/docs\/tokenizers\/python\/latest\/index.html",">","`","_",":",":",":","\ud83e\udd17","Tokenizers","provides","an","implementation","of","today","\u2019","s","most","used","tokenizers",",","with","a","focus","on","performance","and","versatility",".","Use","of","Hugging","Face","tokenizers","for","training","on","custom","data","using","an","sklearn","compatible","API",".","Examples","--","--","--","--",">",">",">","#","train","a","BERT","tokenizer","from","scratch",">",">",">","from","tokenizers","import","Tokenizer",">",">",">","from","tokenizers.models","import","WordPiece",">",">",">","from","tokenizers","import","normalizers",">",">",">","from","tokenizers.normalizers","import","Lowercase",",","NFD",",","StripAccents",">",">",">","from","tokenizers.pre_tokenizers","import","Whitespace",">",">",">","from","tokenizers.processors","import","TemplateProcessing",">",">",">","bert_tokenizer","=","Tokenizer","(","WordPiece","(","unk_token=","''","[","UNK","]","''",")",")",">",">",">","normalizer","=","normalizers.Sequence","(","[","NFD","(",")",",","Lowercase","(",")",",","StripAccents","(",")","]",")",">",">",">","pre_tokenizer","=","Whitespace","(",")",">",">",">","post_processor","=","TemplateProcessing","(","...","single=","''","[","CLS","]","$","A","[","SEP","]","''",",","...","pair=","''","[","CLS","]","$","A","[","SEP","]","$","B:1","[","SEP","]",":1","''",",","...","special_tokens=","[","...","(","``","[","CLS","]","''",",","1",")",",","...","(","``","[","SEP","]","''",",","2",")",",","...","]",",","...",")",">",">",">","from","skorch.hf","import","HuggingfaceTokenizer",">",">",">","hf_tokenizer","=","HuggingfaceTokenizer","(","...","tokenizer=bert_tokenizer",",","...","pre_tokenizer=pre_tokenizer",",","...","post_processor=post_processor",",","...","trainer__vocab_size=30522",",","...","trainer__special_tokens=","[","``","[","UNK","]","''",",","``","[","CLS","]","''",",","``","[","SEP","]","''",",","``","[","PAD","]","''",",","``","[","MASK","]","''","]",",","...",")",">",">",">","data","=","[","'hello","there","'",",","'this","is","a","text","'","]",">",">",">","hf_tokenizer.fit","(","data",")",">",">",">","hf_tokenizer.transform","(","data",")","In","general",",","you","can","pass","both","initialized","objects","and","uninitialized","objects","as","parameters",":","..","code",":",":","python","#","initialized","HuggingfaceTokenizer","(","tokenizer=Tokenizer","(","model=WordPiece","(",")",")",")","#","uninitialized","HuggingfaceTokenizer","(","tokenizer=Tokenizer",",","model=WordPiece",")","Both","approaches","work","equally","well","and","allow","you","to",",","for","instance",",","grid","search","on","the","tokenizer","parameters",".","However",",","it","is","recommended","*","not","*","to","pass","an","initialized","trainer",".","This","is","because","the","trainer","will","then","be","saved","as","an","attribute","on","the","object",",","which","can","be","wasteful",".","Instead",",","it","is","best","to","leave","the","default","``","trainer='auto","'","``",",","which","results","in","the","trainer","being","derived","from","the","model",".","..","note",":",":","If","you","want","to","train","the","``","HuggingfaceTokenizer","``","in","parallel","(","e.g",".","during","a","grid","search",")",",","you","should","probably","set","the","environment","variable","``","TOKENIZERS_PARALLELISM=false","``",".","Otherwise",",","you","may","experience","slow","downs","or","deadlocks",".","Parameters","--","--","--","--","--","tokenizer",":","tokenizers.Tokenizer","The","tokenizer","to","train",".","model",":","tokenizers.models.Model","The","model","represents","the","actual","tokenization","algorithm",",","e.g",".","``","BPE","``",".","trainer",":","tokenizers.trainers.Trainer","or","'auto","'","(","default='auto","'",")","Class","responsible","for","training","the","tokenizer",".","If","'auto","'",",","the","correct","trainer","will","be","inferred","from","the","used","model","using","``","model.get_trainer","(",")","``",".","normalizer",":","tokenizers.normalizers.Normalizer","or","None","(","default=None",")","Optional","normalizer",",","e.g",".","for","casting","the","text","to","lowercase",".","pre_tokenizer",":","tokenizers.pre_tokenizers.PreTokenizer","or","None","(","default=None",")","Optional","pre-tokenization",",","e.g",".","splitting","on","space",".","post_processor",":","tokenizers.processors.PostProcessor","Optional","post-processor",",","mostly","used","to","add","special","tokens","for","BERT","etc",".","max_length",":","int","(","default=256",")","Maximum","number","of","tokens","used","per","sequence",".","return_tensors",":","one","of","None",",","str",",","'pt","'",",","'np","'",",","'tf","'","(","default='pt","'",")","What","type","of","result","values","to","return",".","By","default",",","return","a","padded","and","truncated","(","to","``","max_length","``",")","PyTorch","Tensor",".","Similarly",",","'np","'","results","in","a","padded","and","truncated","numpy","array",".","Tensorflow","tensors","are","not","supported","officially","supported","but","should","also","work",".","If","None","or","str",",","return","a","list","of","lists","instead",".","These","lists","are","not","padded","or","truncated",",","thus","each","row","may","have","different","numbers","of","elements",".","return_attention_mask",":","bool","(","default=True",")","Whether","to","return","the","attention","mask",".","return_token_type_ids",":","bool","(","default=False",")","Whether","to","return","the","token","type","ids",".","return_length",":","bool","(","default=False",")","Whether","to","return","the","length","of","the","encoded","inputs",".","pad_token",":","str","(","default=","'","[","PAD","]","'",")","A","special","token","used","to","make","arrays","of","tokens","the","same","size","for","batching","purpose",".","Will","then","be","ignored","by","attention","mechanisms",".","verbose",":","int","(","default=0",")","Whether","the","tokenizer","should","print","more","information","and","warnings",".","Attributes","--","--","--","--","--","vocabulary_",":","dict","A","mapping","of","terms","to","feature","indices",".","fast_tokenizer_",":","transformers.PreTrainedTokenizerFast","If","you","want","to","extract","the","Hugging","Face","tokenizer","to","use","it","without","skorch",",","use","this","attribute",".","..","_tokenizers",":","https",":","\/\/huggingface.co\/docs\/tokenizers\/python\/latest\/index.html"]
docstr: Return an instance initialized with the given parameters
  
  This is a helper method that deals with several possibilities for a
  component that might need to be initialized:
  
  * It is already an instance that's good to go
  * It is an instance but it needs to be re-initialized
  * It's not an instance and needs to be initialized
  
  For the majority of use cases, this comes down to just comes down to
  just initializing the class with its arguments.
  
  Parameters
  ----------
  instance_or_cls
    The instance or class or callable to be initialized.
  
  kwargs : dict
    The keyword arguments to initialize the instance or class. Can be an
    empty dict.
  
  Returns
  -------
  instance
    The initialized component.
docstr_tok: ["Return","an","instance","initialized","with","the","given","parameters","This","is","a","helper","method","that","deals","with","several","possibilities","for","a","component","that","might","need","to","be","initialized",":","*","It","is","already","an","instance","that","'s","good","to","go","*","It","is","an","instance","but","it","needs","to","be","re-initialized","*","It","'s","not","an","instance","and","needs","to","be","initialized","For","the","majority","of","use","cases",",","this","comes","down","to","just","comes","down","to","just","initializing","the","class","with","its","arguments",".","Parameters","--","--","--","--","--","instance_or_cls","The","instance","or","class","or","callable","to","be","initialized",".","kwargs",":","dict","The","keyword","arguments","to","initialize","the","instance","or","class",".","Can","be","an","empty","dict",".","Returns","--","--","--","-","instance","The","initialized","component","."]
code: def initialized_instance(self, instance_or_cls, kwargs):
      """Return an instance initialized with the given parameters
  
      This is a helper method that deals with several possibilities for a
      component that might need to be initialized:
  
      * It is already an instance that's good to go
      * It is an instance but it needs to be re-initialized
      * It's not an instance and needs to be initialized
  
      For the majority of use cases, this comes down to just comes down to
      just initializing the class with its arguments.
  
      Parameters
      ----------
      instance_or_cls
        The instance or class or callable to be initialized.
  
      kwargs : dict
        The keyword arguments to initialize the instance or class. Can be an
        empty dict.
  
      Returns
      -------
      instance
        The initialized component.
  
      """
      # This whole method is taken from NeuralNet
      if instance_or_cls is None:
          return None
  
      is_init = not isinstance(instance_or_cls, type)
      if is_init and not kwargs:
          return instance_or_cls
      if is_init:
          if self.verbose:
              print(f"Re-initializing {instance_or_cls}")
          return type(instance_or_cls)(**kwargs)
      return instance_or_cls(**kwargs)
  
imports: import io
  import os
  import pathlib
  from copy import deepcopy
  from operator import itemgetter
  import numpy
  import torch
  from sklearn.base import BaseEstimator, TransformerMixin
  from skorch.callbacks import LRScheduler
  from skorch.dataset import unpack_data
  from skorch.utils import check_is_fitted, params_for
label_desc: Use image node_id 3 for calling the HuggingfaceTokenizer obj's underlying member method code with example usage: obj.initialized_instance(instance_or_cls, kwargs) and returns: instance_or_cls, None, instance_or_cls, type