docstr: Args:
      append_eos: if True, each input sentence in the source tokens tensor
          will have an EOS appended to the end.
  
  Returns:
      vocabs: BPE vocab with continuation markers as suffixes to denote
          non-end of word tokens. This is the standard BPE format used in
          fairseq's preprocessing.
      x: input tensor containing numberized source tokens, with EOS at the
          end if append_eos is true
      src_lengths: and source lengths.
docstr_tok: ["Args",":","append_eos",":","if","True",",","each","input","sentence","in","the","source","tokens","tensor","will","have","an","EOS","appended","to","the","end",".","Returns",":","vocabs",":","BPE","vocab","with","continuation","markers","as","suffixes","to","denote","non-end","of","word","tokens",".","This","is","the","standard","BPE","format","used","in","fairseq","'s","preprocessing",".","x",":","input","tensor","containing","numberized","source","tokens",",","with","EOS","at","the","end","if","append_eos","is","true","src_lengths",":","and","source","lengths","."]
code: def _get_test_data_with_bpe_cont_marker(self, append_eos=True):
      """
      Args:
          append_eos: if True, each input sentence in the source tokens tensor
              will have an EOS appended to the end.
  
      Returns:
          vocabs: BPE vocab with continuation markers as suffixes to denote
              non-end of word tokens. This is the standard BPE format used in
              fairseq's preprocessing.
          x: input tensor containing numberized source tokens, with EOS at the
              end if append_eos is true
          src_lengths: and source lengths.
      """
      vocab = Dictionary()
      vocab.add_symbol("he@@")
      vocab.add_symbol("llo")
      vocab.add_symbol("how")
      vocab.add_symbol("are")
      vocab.add_symbol("y@@")
      vocab.add_symbol("ou")
      vocab.add_symbol("n@@")
      vocab.add_symbol("ew")
      vocab.add_symbol("or@@")
      vocab.add_symbol("k")
  
      src_tokens = [
          ["he@@", "llo", "n@@", "ew", "y@@", "or@@", "k"],
          ["how", "are", "y@@", "ou"],
      ]
      x, src_lengths = (
          x,
          src_lengths,
      ) = self._convert_src_tokens_to_tensor(
          vocab=vocab, src_tokens=src_tokens, append_eos=append_eos
      )
      return vocab, x, src_lengths
  
imports: import unittest
  from typing import Dict, List
  import torch
  import tests.utils
  from fairseq import utils
  from fairseq.data import Dictionary, LanguagePairDataset, TransformEosDataset, data_utils, noising
label_desc: Use image node_id 1 for calling the TestDataNoising obj's underlying member method code with example usage: obj._get_test_data_with_bpe_cont_marker(append_eos) and returns: vocab,  x,  src_lengths