class_docstr: External merger will dump the aggregated data into disks when
  memory usage goes above the limit, then merge them together.
  
  This class works as follows:
  
  - It repeatedly combine the items and save them in one dict in
    memory.
  
  - When the used memory goes above memory limit, it will split
    the combined data into partitions by hash code, dump them
    into disk, one file per partition.
  
  - Then it goes through the rest of the iterator, combine items
    into different dict by hash. Until the used memory goes over
    memory limit, it dump all the dicts into disks, one file per
    dict. Repeat this again until combine all the items.
  
  - Before return any items, it will load each partition and
    combine them separately. Yield them before loading next
    partition.
  
  - During loading a partition, if the memory goes over limit,
    it will partition the loaded data and dump them into disks
    and load them partition by partition again.
  
  `data` and `pdata` are used to hold the merged items in memory.
  At first, all the data are merged into `data`. Once the used
  memory goes over limit, the items in `data` are dumped into
  disks, `data` will be cleared, all rest of items will be merged
  into `pdata` and then dumped into disks. Before returning, all
  the items in `pdata` will be dumped into disks.
  
  Finally, if any items were spilled into disks, each partition
  will be merged into `data` and be yielded, then cleared.
  
  Examples
  --------
  >>> agg = SimpleAggregator(lambda x, y: x + y)
  >>> merger = ExternalMerger(agg, 10)
  >>> N = 10000
  >>> merger.mergeValues(zip(range(N), range(N)))
  >>> assert merger.spills > 0
  >>> sum(v for k,v in merger.items())
  49995000
  
  >>> merger = ExternalMerger(agg, 10)
  >>> merger.mergeCombiners(zip(range(N), range(N)))
  >>> assert merger.spills > 0
  >>> sum(v for k,v in merger.items())
  49995000
class_docstr_tok: ["External","merger","will","dump","the","aggregated","data","into","disks","when","memory","usage","goes","above","the","limit",",","then","merge","them","together",".","This","class","works","as","follows",":","-","It","repeatedly","combine","the","items","and","save","them","in","one","dict","in","memory",".","-","When","the","used","memory","goes","above","memory","limit",",","it","will","split","the","combined","data","into","partitions","by","hash","code",",","dump","them","into","disk",",","one","file","per","partition",".","-","Then","it","goes","through","the","rest","of","the","iterator",",","combine","items","into","different","dict","by","hash",".","Until","the","used","memory","goes","over","memory","limit",",","it","dump","all","the","dicts","into","disks",",","one","file","per","dict",".","Repeat","this","again","until","combine","all","the","items",".","-","Before","return","any","items",",","it","will","load","each","partition","and","combine","them","separately",".","Yield","them","before","loading","next","partition",".","-","During","loading","a","partition",",","if","the","memory","goes","over","limit",",","it","will","partition","the","loaded","data","and","dump","them","into","disks","and","load","them","partition","by","partition","again",".","`","data","`","and","`","pdata","`","are","used","to","hold","the","merged","items","in","memory",".","At","first",",","all","the","data","are","merged","into","`","data","`",".","Once","the","used","memory","goes","over","limit",",","the","items","in","`","data","`","are","dumped","into","disks",",","`","data","`","will","be","cleared",",","all","rest","of","items","will","be","merged","into","`","pdata","`","and","then","dumped","into","disks",".","Before","returning",",","all","the","items","in","`","pdata","`","will","be","dumped","into","disks",".","Finally",",","if","any","items","were","spilled","into","disks",",","each","partition","will","be","merged","into","`","data","`","and","be","yielded",",","then","cleared",".","Examples","--","--","--","--",">",">",">","agg","=","SimpleAggregator","(","lambda","x",",","y",":","x","+","y",")",">",">",">","merger","=","ExternalMerger","(","agg",",","10",")",">",">",">","N","=","10000",">",">",">","merger.mergeValues","(","zip","(","range","(","N",")",",","range","(","N",")",")",")",">",">",">","assert","merger.spills",">","0",">",">",">","sum","(","v","for","k",",","v","in","merger.items","(",")",")","49995000",">",">",">","merger","=","ExternalMerger","(","agg",",","10",")",">",">",">","merger.mergeCombiners","(","zip","(","range","(","N",")",",","range","(","N",")",")",")",">",">",">","assert","merger.spills",">","0",">",">",">","sum","(","v","for","k",",","v","in","merger.items","(",")",")","49995000"]
docstr: Return all merged items as iterator
docstr_tok: ["Return","all","merged","items","as","iterator"]
code: def items(self):
      """Return all merged items as iterator"""
      if not self.pdata and not self.spills:
          return iter(self.data.items())
      return self._external_items()
  
imports: import os
  import platform
  import shutil
  import warnings
  import gc
  import itertools
  import operator
  import random
  import sys
  import heapq
  from pyspark.serializers import BatchedSerializer, CPickleSerializer, FlattenedValuesSerializer, CompressedSerializer, AutoBatchedSerializer
  from pyspark.util import fail_on_stopiteration
label_desc: Use image node_id 9 for calling the ExternalMerger obj's underlying member method code with example usage: obj.items() and returns: self, iter