docstr: Args:
      append_eos: if True, each input sentence in the source tokens tensor
          will have an EOS appended to the end.
  
  Returns:
      vocabs: BPE vocab with end-of-word markers as suffixes to denote
          tokens at the end of a word. This is an alternative to fairseq's
          standard preprocessing framework and is not generally supported
          within fairseq.
      x: input tensor containing numberized source tokens, with EOS at the
          end if append_eos is true
      src_lengths: and source lengths.
docstr_tok: ["Args",":","append_eos",":","if","True",",","each","input","sentence","in","the","source","tokens","tensor","will","have","an","EOS","appended","to","the","end",".","Returns",":","vocabs",":","BPE","vocab","with","end-of-word","markers","as","suffixes","to","denote","tokens","at","the","end","of","a","word",".","This","is","an","alternative","to","fairseq's","standard","preprocessing","framework","and","is","not","generally","supported","within","fairseq",".","x",":","input","tensor","containing","numberized","source","tokens",",","with","EOS","at","the","end","if","append_eos","is","true","src_lengths",":","and","source","lengths","."]
code: def _get_test_data_with_bpe_end_marker(self, append_eos=True):
      """
      Args:
          append_eos: if True, each input sentence in the source tokens tensor
              will have an EOS appended to the end.
  
      Returns:
          vocabs: BPE vocab with end-of-word markers as suffixes to denote
              tokens at the end of a word. This is an alternative to fairseq's
              standard preprocessing framework and is not generally supported
              within fairseq.
          x: input tensor containing numberized source tokens, with EOS at the
              end if append_eos is true
          src_lengths: and source lengths.
      """
      vocab = Dictionary()
      vocab.add_symbol("he")
      vocab.add_symbol("llo_EOW")
      vocab.add_symbol("how_EOW")
      vocab.add_symbol("are_EOW")
      vocab.add_symbol("y")
      vocab.add_symbol("ou_EOW")
      vocab.add_symbol("n")
      vocab.add_symbol("ew_EOW")
      vocab.add_symbol("or")
      vocab.add_symbol("k_EOW")
  
      src_tokens = [
          ["he", "llo_EOW", "n", "ew_EOW", "y", "or", "k_EOW"],
          ["how_EOW", "are_EOW", "y", "ou_EOW"],
      ]
      x, src_lengths = (
          x,
          src_lengths,
      ) = self._convert_src_tokens_to_tensor(
          vocab=vocab, src_tokens=src_tokens, append_eos=append_eos
      )
      return vocab, x, src_lengths
  
imports: import unittest
  from typing import Dict, List
  import torch
  import tests.utils
  from fairseq import utils
  from fairseq.data import Dictionary, LanguagePairDataset, TransformEosDataset, data_utils, noising
label_desc: Use image node_id 2 for calling the TestDataNoising obj's underlying member method code with example usage: obj._get_test_data_with_bpe_end_marker(append_eos) and returns: vocab,  x,  src_lengths