class_docstr: Abstracts the dataset used to train seq2seq models.
  The class will process the documents that are located in the specified
  folder. The preprocessing will work on any document that is reasonably
  formatted. On the CNN/DailyMail dataset it will extract both the story
  and the summary.
  CNN/Daily News:
  The CNN/Daily News raw datasets are downloaded from [1]. The stories are
  stored in different files; the summary appears at the end of the story as
  sentences that are prefixed by the special `@highlight` line. To process
  the data, untar both datasets in the same folder, and pass the path to this
  folder as the "data_dir argument. The formatting code was inspired by [2].
  [1] https://cs.nyu.edu/~kcho/
  [2] https://github.com/abisee/cnn-dailymail/
class_docstr_tok: ["Abstracts","the","dataset","used","to","train","seq2seq","models",".","The","class","will","process","the","documents","that","are","located","in","the","specified","folder",".","The","preprocessing","will","work","on","any","document","that","is","reasonably","formatted",".","On","the","CNN\/DailyMail","dataset","it","will","extract","both","the","story","and","the","summary",".","CNN\/Daily","News",":","The","CNN\/Daily","News","raw","datasets","are","downloaded","from","[","1","]",".","The","stories","are","stored","in","different","files",";","the","summary","appears","at","the","end","of","the","story","as","sentences","that","are","prefixed","by","the","special","`","@","highlight","`","line",".","To","process","the","data",",","untar","both","datasets","in","the","same","folder",",","and","pass","the","path","to","this","folder","as","the","``","data_dir","argument",".","The","formatting","code","was","inspired","by","[","2","]",".","[","1","]","https",":","\/\/cs.nyu.edu\/~kcho\/","[","2","]","https",":","\/\/github.com\/abisee\/cnn-dailymail\/"]
code: def __getitem__(self, idx):
      document_path = self.documents[idx]
      document_name = document_path.split("/")[-1]
      with open(document_path, encoding="utf-8") as source:
          raw_doc = source.read()
          doc_lines = process_document(raw_doc)
      return document_name, doc_lines, []
  
imports: import re
  import html
  import logging
  import pandas
  import os
  import random
  import torch
  from pathlib import Path
  import pickle
  import shutil
  from collections import deque, namedtuple
  from torch.utils.data import Dataset, DataLoader, SequentialSampler
  from tokenizers import BertWordPieceTokenizer
  from transformers import BertTokenizer
label_desc: Use image node_id 3 for calling the SummarizationDataset obj's underlying member method code with example usage: obj.__getitem__(idx) and returns: document_name,  doc_lines,  list