docstr: Given a tokenized text and the starting index of a word, the function selects the entire word.
  Note that a word is composed of multiple tokens (e.g., ``word = [head_token tail_token_1
  tail_token_2 ... tail_token_k]``). The tail tokens can be identified based on the
  presence/absence of `SUBWORD_PREFIX`. See :py:meth:`alibi.utils.lang_model.LanguageModel.is_subword_prefix`
  for more details.
  
  Parameters
  ----------
  tokenized_text
      Tokenized text.
  start_idx
      Starting index of a word.
  punctuation
      String of punctuation to be considered. If it encounters a token
      composed only of characters in `punctuation` it terminates the search.
  
  Returns
  -------
  The word obtained by concatenation ``[head_token tail_token_1 tail_token_2 ... tail_token_k]``.
docstr_tok: ["Given","a","tokenized","text","and","the","starting","index","of","a","word",",","the","function","selects","the","entire","word",".","Note","that","a","word","is","composed","of","multiple","tokens","(","e.g.",",","``","word","=","[","head_token","tail_token_1","tail_token_2","...","tail_token_k","]","``",")",".","The","tail","tokens","can","be","identified","based","on","the","presence\/absence","of","`","SUBWORD_PREFIX","`",".","See",":","py",":","meth",":","`","alibi.utils.lang_model.LanguageModel.is_subword_prefix","`","for","more","details",".","Parameters","--","--","--","--","--","tokenized_text","Tokenized","text",".","start_idx","Starting","index","of","a","word",".","punctuation","String","of","punctuation","to","be","considered",".","If","it","encounters","a","token","composed","only","of","characters","in","`","punctuation","`","it","terminates","the","search",".","Returns","--","--","--","-","The","word","obtained","by","concatenation","``","[","head_token","tail_token_1","tail_token_2","...","tail_token_k","]","``","."]
code: def select_word(
      self, tokenized_text: List[str], start_idx: int, punctuation: str
  ) -> str:
      """
      Given a tokenized text and the starting index of a word, the function selects the entire word.
      Note that a word is composed of multiple tokens (e.g., ``word = [head_token tail_token_1
      tail_token_2 ... tail_token_k]``). The tail tokens can be identified based on the
      presence/absence of `SUBWORD_PREFIX`. See :py:meth:`alibi.utils.lang_model.LanguageModel.is_subword_prefix`
      for more details.
  
      Parameters
      ----------
      tokenized_text
          Tokenized text.
      start_idx
          Starting index of a word.
      punctuation
          String of punctuation to be considered. If it encounters a token
          composed only of characters in `punctuation` it terminates the search.
  
      Returns
      -------
      The word obtained by concatenation ``[head_token tail_token_1 tail_token_2 ... tail_token_k]``.
      """
      # define the ending index
      end_idx = start_idx + 1
  
      while end_idx < len(tokenized_text):
          # The second condition is necessary for models like Roberta.
          # If the second condition is not included, it can select words like: `word,` instead of `word`
          if (
              not self.is_subword_prefix(tokenized_text[end_idx])
          ) or self.is_punctuation(
              tokenized_text[end_idx], punctuation
          ):
              break
  
          end_idx += 1
  
      # convert the tokens into a string
      word = self.tokenizer.convert_tokens_to_string(
          tokenized_text[start_idx:end_idx]
      )
      return word
  
imports: import abc
  import numpy
  from pathlib import Path
  from typing import Any, Callable, List, Optional, Tuple, Union
  import tensorflow
  import transformers
  from transformers import TFAutoModelForMaskedLM, AutoTokenizer
label_desc: Use image node_id 5 for calling the LanguageModel obj's underlying member method code with example usage: obj.select_word(tokenized_text, start_idx, punctuation) and returns: word