class_docstr: Streaming Data Silo loads and preprocesses datasets in parallel to the model training.
  
  The samples are lazily created from the input file and batches are yielded on-the-fly when required during training.
  This is useful if you:
  - work with large datasets that don't fit in memory
  - want to save time (by not preprocessing the entire dataset before starting training)
  
  For optimal training performance and efficient utilization of shiny GPUs, the pipeline always keeps a few
  pre-computed batches ready to avoid any waiting time when a batch is requested during training.
  
  To parallelize the creation of batches, PyTorch DataLoader provide an option to use
  multiple workers that utilize the available CPU cores and ensure enough pre-computed batches.
class_docstr_tok: ["Streaming","Data","Silo","loads","and","preprocesses","datasets","in","parallel","to","the","model","training",".","The","samples","are","lazily","created","from","the","input","file","and","batches","are","yielded","on-the-fly","when","required","during","training",".","This","is","useful","if","you",":","-","work","with","large","datasets","that","do","n't","fit","in","memory","-","want","to","save","time","(","by","not","preprocessing","the","entire","dataset","before","starting","training",")","For","optimal","training","performance","and","efficient","utilization","of","shiny","GPUs",",","the","pipeline","always","keeps","a","few","pre-computed","batches","ready","to","avoid","any","waiting","time","when","a","batch","is","requested","during","training",".","To","parallelize","the","creation","of","batches",",","PyTorch","DataLoader","provide","an","option","to","use","multiple","workers","that","utilize","the","available","CPU","cores","and","ensure","enough","pre-computed","batches","."]
docstr: Returns a new instance of dataloader for the given dataset.
  
  The dataloader lazily yields from Iterable DataSets. After a complete iteration
  over the input data, the generators gets exhausted. So, for instance, in the 
  case of model training, a new train dataloader must be used for each train epoch.
  
  :param dataset_name: 'train', 'dev', or 'test' set.
  :type dataset_name: str
docstr_tok: ["Returns","a","new","instance","of","dataloader","for","the","given","dataset",".","The","dataloader","lazily","yields","from","Iterable","DataSets",".","After","a","complete","iteration","over","the","input","data",",","the","generators","gets","exhausted",".","So",",","for","instance",",","in","the","case","of","model","training",",","a","new","train","dataloader","must","be","used","for","each","train","epoch",".",":","param","dataset_name",":","'train","'",",","'dev","'",",","or","'test","'","set",".",":","type","dataset_name",":","str"]
code: def get_data_loader(self, dataset_name):
      """
      Returns a new instance of dataloader for the given dataset.
  
      The dataloader lazily yields from Iterable DataSets. After a complete iteration
      over the input data, the generators gets exhausted. So, for instance, in the
      case of model training, a new train dataloader must be used for each train epoch.
  
      :param dataset_name: 'train', 'dev', or 'test' set.
      :type dataset_name: str
      """
      filename = None
      if dataset_name == "train":
          filename = self.processor.train_filename
      elif dataset_name == "dev":
          if self.processor.dev_split > 0.0:
              raise NotImplemented(
                  "StreamingDataSilo does not have dev_split implemented. "
                  "To use dev data, supply a dev filename when creating the Processor."
              )
          elif self.processor.dev_filename:
              filename = self.processor.dev_filename
      elif dataset_name == "test":
          if self.processor.test_filename:
              filename = self.processor.test_filename
  
      if not filename:
          return None
  
      #  Batching:
      #
      #  The model Trainer is passed a PyTorch DataLoader instance that yields dataset batches for training.
      #
      #  By default, the PyTorch DataLoader prefetch (2 * num_workers) samples. However, given the higher
      #  batch sizes(usually >64) for model training, the default prefetch is not sufficient to keep the
      #  model Training saturated with datasets.
      #
      #  As a workaround, we yield batches of samples instead of yielding individual samples. The DataLoader
      #  can then prefetch (2 * num_workers) number of batches of samples.
      #
      #  Since the batching is now handled within _StreamingDataSet, we disable the batching on DataLoader side
      #  by initializing the data loader with batch_size as 1.
  
      if isinstance(filename, Path) and filename.is_dir():
          filepath = filename
      else:
          filepath = self.processor.data_dir / filename
  
      data_set = _StreamingDataSet(
          processor=self.processor,
          filepath=filepath,
          batch_size=self.batch_size,
          dataloader_workers=self.dataloader_workers,
          distributed=self.distributed,
      )
  
      data_loader = NamedDataLoader(
          dataset=data_set,
          batch_size=1,
          num_workers=self.dataloader_workers,
          pin_memory=True,
      )
      return data_loader
  
imports: import copy
  import logging
  import torch.multiprocessing
  from contextlib import ExitStack
  from functools import partial
  import random
  from pathlib import Path
  from itertools import chain, groupby
  import numpy
  from sklearn.utils.class_weight import compute_class_weight
  from torch.utils.data import Dataset, Subset, IterableDataset
  from torch.utils.data.distributed import DistributedSampler
  from torch.utils.data.sampler import RandomSampler, SequentialSampler
  import torch
  from sklearn.model_selection import StratifiedKFold, KFold, ShuffleSplit, StratifiedShuffleSplit
  from tqdm import tqdm
  from farm.data_handler.dataloader import NamedDataLoader
  from farm.data_handler.processor import Processor, BertStyleLMProcessor
  from farm.data_handler.utils import grouper
  from farm.data_handler.dataset import ConcatTensorDataset
  from farm.modeling.tokenization import EmbeddingTokenizer
  from farm.utils import MLFlowLogger
  from farm.utils import log_ascii_workers, calc_chunksize
  from farm.utils import get_dict_checksum
  from farm.visual.ascii.images import TRACTOR_SMALL
label_desc: Use image node_id 2 for calling the StreamingDataSilo obj's underlying member method code with example usage: obj.get_data_loader(dataset_name) and returns: data_loader, None