class_docstr: Base class for yet to train and pretrained tokenizers
  
  Implements the ``vocabulary_`` attribute and the methods
  ``get_feature_names``, ``transform``, and ``inverse_transform``.
  
  Subclasses should implement the ``fit``  method.
class_docstr_tok: ["Base","class","for","yet","to","train","and","pretrained","tokenizers","Implements","the","``","vocabulary_","``","attribute","and","the","methods","``","get_feature_names","``",",","``","transform","``",",","and","``","inverse_transform","``",".","Subclasses","should","implement","the","``","fit","``","method","."]
docstr: Convenience method to use the trained tokenizer for tokenization
  
  The input text is not encoded into integers, instead the strings are
  kept.
  
  Use this method if you're mainly interested in splitting the text into
  tokens using the trained Hugging Face tokenizer.
  
  Parameters
  ----------
  X : iterable of str
    A list/array of strings or an iterable which generates either strings.
  
  kwargs : dict
    Additional arguments, passed directly to the ``decode`` method of the
    tokenizer, e.g. ``skip_special_tokens=True``.
  
  Returns
  -------
  Xt : np.ndarray
    2d array containing, in each row, an array of strings corresponding to
    the tokenized input text.
docstr_tok: ["Convenience","method","to","use","the","trained","tokenizer","for","tokenization","The","input","text","is","not","encoded","into","integers",",","instead","the","strings","are","kept",".","Use","this","method","if","you","'re","mainly","interested","in","splitting","the","text","into","tokens","using","the","trained","Hugging","Face","tokenizer",".","Parameters","--","--","--","--","--","X",":","iterable","of","str","A","list\/array","of","strings","or","an","iterable","which","generates","either","strings",".","kwargs",":","dict","Additional","arguments",",","passed","directly","to","the","``","decode","``","method","of","the","tokenizer",",","e.g",".","``","skip_special_tokens=True","``",".","Returns","--","--","--","-","Xt",":","np.ndarray","2d","array","containing",",","in","each","row",",","an","array","of","strings","corresponding","to","the","tokenized","input","text","."]
code: def tokenize(self, X, **kwargs):
      """Convenience method to use the trained tokenizer for tokenization
  
      The input text is not encoded into integers, instead the strings are
      kept.
  
      Use this method if you're mainly interested in splitting the text into
      tokens using the trained Hugging Face tokenizer.
  
      Parameters
      ----------
      X : iterable of str
        A list/array of strings or an iterable which generates either strings.
  
      kwargs : dict
        Additional arguments, passed directly to the ``decode`` method of the
        tokenizer, e.g. ``skip_special_tokens=True``.
  
      Returns
      -------
      Xt : np.ndarray
        2d array containing, in each row, an array of strings corresponding to
        the tokenized input text.
  
      """
      check_is_fitted(self, ["fast_tokenizer_"])
  
      encoded = self.transform(X)
      tokenizer = self.fast_tokenizer_
      Xt = []
      for token_ids in encoded["input_ids"]:
          tokens = [
              tokenizer.decode(token_id, **kwargs)
              for token_id in token_ids
          ]
          Xt.append(tokens)
      return np.asarray(Xt)
  
imports: import io
  import os
  import pathlib
  from copy import deepcopy
  from operator import itemgetter
  import numpy
  import torch
  from sklearn.base import BaseEstimator, TransformerMixin
  from skorch.callbacks import LRScheduler
  from skorch.dataset import unpack_data
  from skorch.utils import check_is_fitted, params_for
label_desc: Use image node_id 7 for calling the _HuggingfaceTokenizerBase obj's underlying member method code with example usage: obj.tokenize(X) and returns: np