class_docstr: A similar Dataset to the AudioLabelDataSet, but which loads tarred audio files.
  
  Accepts a single comma-separated JSON manifest file (in the same style as for the AudioToSpeechLabelDataset),
  as well as the path(s) to the tarball(s) containing the wav files. Each line of the manifest should
  contain the information for one audio file, including at least the label and name of the audio
  file within the tarball.
  
  Valid formats for the audio_tar_filepaths argument include:
  (1) a single string that can be brace-expanded, e.g. 'path/to/audio.tar' or 'path/to/audio_{1..100}.tar.gz', or
  (2) a list of file paths that will not be brace-expanded, e.g. ['audio_1.tar', 'audio_2.tar', ...].
  
  Note: For brace expansion in (1), there may be cases where `{x..y}` syntax cannot be used due to shell interference.
  This occurs most commonly inside SLURM scripts. Therefore we provide a few equivalent replacements.
  Supported opening braces - { <=> (, [, < and the special tag _OP_.
  Supported closing braces - } <=> ), ], > and the special tag _CL_.
  For SLURM based tasks, we suggest the use of the special tags for ease of use.
  
  See the documentation for more information about accepted data and input formats.
  
  If using multiple processes the number of shards should be divisible by the number of workers to ensure an
  even split among workers. If it is not divisible, logging will give a warning but training will proceed.
  In addition, if using mutiprocessing, each shard MUST HAVE THE SAME NUMBER OF ENTRIES after filtering
  is applied. We currently do not check for this, but your program may hang if the shards are uneven!
  
  Notice that a few arguments are different from the AudioLabelDataSet; for example, shuffle (bool) has been
  replaced by shuffle_n (int).
  
  Additionally, please note that the len() of this DataLayer is assumed to be the length of the manifest
  after filtering. An incorrect manifest length may lead to some DataLoader issues down the line.
  
  Args:
      audio_tar_filepaths: Either a list of audio tarball filepaths, or a
          string (can be brace-expandable).
      manifest_filepath (str): Path to the manifest.
      labels (list): Dataset parameter.
          List of target classes that can be output by the speaker recognition model.
      featurizer
      shuffle_n (int): How many samples to look ahead and load to be shuffled.
          See WebDataset documentation for more details.
          Defaults to 0.
      min_duration (float): Dataset parameter.
          All training files which have a duration less than min_duration
          are dropped. Note: Duration is read from the manifest JSON.
          Defaults to 0.1.
      max_duration (float): Dataset parameter.
          All training files which have a duration more than max_duration
          are dropped. Note: Duration is read from the manifest JSON.
          Defaults to None.
      trim(bool): Whether to use trim silence from beginning and end
          of audio signal using librosa.effects.trim().
          Defaults to False.
      window_length_in_sec (float): length of slice/window (in seconds) # Pass this only for speaker recognition and VAD task
      shift_length_in_sec (float): amount of shift of window for generating the frame for VAD task. in a batch # Pass this only for VAD task during inference.
      normalize_audio (bool): Whether to normalize audio signal. Defaults to False.
      shard_strategy (str): Tarred dataset shard distribution strategy chosen as a str value during ddp.
          -   `scatter`: The default shard strategy applied by WebDataset, where each node gets
              a unique set of shards, which are permanently pre-allocated and never changed at runtime.
          -   `replicate`: Optional shard strategy, where each node gets all of the set of shards
              available in the tarred dataset, which are permanently pre-allocated and never changed at runtime.
              The benefit of replication is that it allows each node to sample data points from the entire
              dataset independently of other nodes, and reduces dependence on the value of `shuffle_n`.
  
              .. warning::
                  Replicated strategy allows every node to sample the entire set of available tarfiles,
                  and therefore more than one node may sample the same tarfile, and even sample the same
                  data points! As such, there is no assured guarantee that all samples in the dataset will be
                  sampled at least once during 1 epoch. Scattered strategy, on the other hand, on specific
                  occasions (when the number of shards is not divisible with ``world_size``), will not sample
                  the entire dataset. For these reasons it is not advisable to use tarred datasets as validation
                  or test datasets.
      global_rank (int): Worker rank, used for partitioning shards. Defaults to 0.
      world_size (int): Total number of processes, used for partitioning shards. Defaults to 0.
      is_regression_task (bool): Whether it is a regression task. Defualts to False.
class_docstr_tok: ["A","similar","Dataset","to","the","AudioLabelDataSet",",","but","which","loads","tarred","audio","files",".","Accepts","a","single","comma-separated","JSON","manifest","file","(","in","the","same","style","as","for","the","AudioToSpeechLabelDataset",")",",","as","well","as","the","path","(","s",")","to","the","tarball","(","s",")","containing","the","wav","files",".","Each","line","of","the","manifest","should","contain","the","information","for","one","audio","file",",","including","at","least","the","label","and","name","of","the","audio","file","within","the","tarball",".","Valid","formats","for","the","audio_tar_filepaths","argument","include",":","(","1",")","a","single","string","that","can","be","brace-expanded",",","e.g",".","'path\/to\/audio.tar","'","or","'path\/to\/audio_","{","1","..","100","}",".tar.gz","'",",","or","(","2",")","a","list","of","file","paths","that","will","not","be","brace-expanded",",","e.g",".","[","'audio_1.tar","'",",","'audio_2.tar","'",",","...","]",".","Note",":","For","brace","expansion","in","(","1",")",",","there","may","be","cases","where","`","{","x","..","y","}","`","syntax","can","not","be","used","due","to","shell","interference",".","This","occurs","most","commonly","inside","SLURM","scripts",".","Therefore","we","provide","a","few","equivalent","replacements",".","Supported","opening","braces","-","{","<","=",">","(",",","[",",","<","and","the","special","tag","_OP_",".","Supported","closing","braces","-","}","<","=",">",")",",","]",",",">","and","the","special","tag","_CL_",".","For","SLURM","based","tasks",",","we","suggest","the","use","of","the","special","tags","for","ease","of","use",".","See","the","documentation","for","more","information","about","accepted","data","and","input","formats",".","If","using","multiple","processes","the","number","of","shards","should","be","divisible","by","the","number","of","workers","to","ensure","an","even","split","among","workers",".","If","it","is","not","divisible",",","logging","will","give","a","warning","but","training","will","proceed",".","In","addition",",","if","using","mutiprocessing",",","each","shard","MUST","HAVE","THE","SAME","NUMBER","OF","ENTRIES","after","filtering","is","applied",".","We","currently","do","not","check","for","this",",","but","your","program","may","hang","if","the","shards","are","uneven","!","Notice","that","a","few","arguments","are","different","from","the","AudioLabelDataSet",";","for","example",",","shuffle","(","bool",")","has","been","replaced","by","shuffle_n","(","int",")",".","Additionally",",","please","note","that","the","len","(",")","of","this","DataLayer","is","assumed","to","be","the","length","of","the","manifest","after","filtering",".","An","incorrect","manifest","length","may","lead","to","some","DataLoader","issues","down","the","line",".","Args",":","audio_tar_filepaths",":","Either","a","list","of","audio","tarball","filepaths",",","or","a","string","(","can","be","brace-expandable",")",".","manifest_filepath","(","str",")",":","Path","to","the","manifest",".","labels","(","list",")",":","Dataset","parameter",".","List","of","target","classes","that","can","be","output","by","the","speaker","recognition","model",".","featurizer","shuffle_n","(","int",")",":","How","many","samples","to","look","ahead","and","load","to","be","shuffled",".","See","WebDataset","documentation","for","more","details",".","Defaults","to","0.","min_duration","(","float",")",":","Dataset","parameter",".","All","training","files","which","have","a","duration","less","than","min_duration","are","dropped",".","Note",":","Duration","is","read","from","the","manifest","JSON",".","Defaults","to","0.1.","max_duration","(","float",")",":","Dataset","parameter",".","All","training","files","which","have","a","duration","more","than","max_duration","are","dropped",".","Note",":","Duration","is","read","from","the","manifest","JSON",".","Defaults","to","None",".","trim","(","bool",")",":","Whether","to","use","trim","silence","from","beginning","and","end","of","audio","signal","using","librosa.effects.trim","(",")",".","Defaults","to","False",".","window_length_in_sec","(","float",")",":","length","of","slice\/window","(","in","seconds",")","#","Pass","this","only","for","speaker","recognition","and","VAD","task","shift_length_in_sec","(","float",")",":","amount","of","shift","of","window","for","generating","the","frame","for","VAD","task",".","in","a","batch","#","Pass","this","only","for","VAD","task","during","inference",".","normalize_audio","(","bool",")",":","Whether","to","normalize","audio","signal",".","Defaults","to","False",".","shard_strategy","(","str",")",":","Tarred","dataset","shard","distribution","strategy","chosen","as","a","str","value","during","ddp",".","-","`","scatter","`",":","The","default","shard","strategy","applied","by","WebDataset",",","where","each","node","gets","a","unique","set","of","shards",",","which","are","permanently","pre-allocated","and","never","changed","at","runtime",".","-","`","replicate","`",":","Optional","shard","strategy",",","where","each","node","gets","all","of","the","set","of","shards","available","in","the","tarred","dataset",",","which","are","permanently","pre-allocated","and","never","changed","at","runtime",".","The","benefit","of","replication","is","that","it","allows","each","node","to","sample","data","points","from","the","entire","dataset","independently","of","other","nodes",",","and","reduces","dependence","on","the","value","of","`","shuffle_n","`",".","..","warning",":",":","Replicated","strategy","allows","every","node","to","sample","the","entire","set","of","available","tarfiles",",","and","therefore","more","than","one","node","may","sample","the","same","tarfile",",","and","even","sample","the","same","data","points","!","As","such",",","there","is","no","assured","guarantee","that","all","samples","in","the","dataset","will","be","sampled","at","least","once","during","1","epoch",".","Scattered","strategy",",","on","the","other","hand",",","on","specific","occasions","(","when","the","number","of","shards","is","not","divisible","with","``","world_size","``",")",",","will","not","sample","the","entire","dataset",".","For","these","reasons","it","is","not","advisable","to","use","tarred","datasets","as","validation","or","test","datasets",".","global_rank","(","int",")",":","Worker","rank",",","used","for","partitioning","shards",".","Defaults","to","0.","world_size","(","int",")",":","Total","number","of","processes",",","used","for","partitioning","shards",".","Defaults","to","0.","is_regression_task","(","bool",")",":","Whether","it","is","a","regression","task",".","Defualts","to","False","."]
docstr: This function is used to remove samples that have been filtered out by ASRSpeechLabel already.
  Otherwise, we would get a KeyError as _build_sample attempts to find the manifest entry for a sample
  that was filtered out (e.g. for duration).
  Note that if using multi-GPU training, filtering may lead to an imbalance in samples in each shard,
  which may make your code hang as one process will finish before the other.
docstr_tok: ["This","function","is","used","to","remove","samples","that","have","been","filtered","out","by","ASRSpeechLabel","already",".","Otherwise",",","we","would","get","a","KeyError","as","_build_sample","attempts","to","find","the","manifest","entry","for","a","sample","that","was","filtered","out","(","e.g",".","for","duration",")",".","Note","that","if","using","multi-GPU","training",",","filtering","may","lead","to","an","imbalance","in","samples","in","each","shard",",","which","may","make","your","code","hang","as","one","process","will","finish","before","the","other","."]
code: def _filter(self, iterator):
      """This function is used to remove samples that have been filtered out by ASRSpeechLabel already.
      Otherwise, we would get a KeyError as _build_sample attempts to find the manifest entry for a sample
      that was filtered out (e.g. for duration).
      Note that if using multi-GPU training, filtering may lead to an imbalance in samples in each shard,
      which may make your code hang as one process will finish before the other.
      """
  
      class TarredAudioFilter:
          def __init__(self, collection, file_occurence):
              self.iterator = iterator
              self.collection = collection
              self.file_occurence = file_occurence
              self._iterable = self._internal_generator()
  
          def __iter__(self):
              self._iterable = self._internal_generator()
              return self
  
          def __next__(self):
              try:
                  values = next(self._iterable)
              except StopIteration:
                  # reset generator
                  self._iterable = self._internal_generator()
                  values = next(self._iterable)
  
              return values
  
          def _internal_generator(self):
              """
              WebDataset requires an Iterator, but we require an iterable that yields 1-or-more
              values per value inside self.iterator.
  
              Therefore wrap the iterator with a generator function that will yield 1-or-more
              values per sample in the iterator.
              """
              for _, tup in enumerate(self.iterator):
                  audio_bytes, audio_filename = tup
  
                  file_id, _ = os.path.splitext(
                      os.path.basename(audio_filename)
                  )
                  if audio_filename in self.file_occurence:
                      for j in range(0, self.file_occurence[file_id]):
                          if j == 0:
                              audio_filename = file_id
                          else:
                              audio_filename = file_id + "-sub" + str(j)
                          yield audio_bytes, audio_filename
  
      return TarredAudioFilter(self.collection, self.file_occurence)
  
imports: import io
  import os
  from typing import Dict, List, Optional, Union
  import torch
  import webdataset
  from nemo.collections.asr.data.audio_to_text import cache_datastore_manifests, expand_sharded_filepaths
  from nemo.collections.asr.parts.preprocessing.features import WaveformFeaturizer
  from nemo.collections.asr.parts.preprocessing.segment import available_formats
  from nemo.collections.common.parts.preprocessing import collections
  from nemo.core.classes import Dataset, IterableDataset
  from nemo.core.neural_types import AudioSignal, LabelsType, LengthsType, NeuralType, RegressionValuesType
  from nemo.utils import logging
label_desc: Use image node_id 2 for calling the _TarredAudioLabelDataset obj's underlying member method code with example usage: obj._filter(iterator) and returns: TarredAudioFilter, self, values