class_docstr: An executor which executes Python functions in the cloud.
  
  :class:`.concurrent.futures.ProcessPoolExecutor` and
  :class:`.concurrent.futures.ThreadPoolExecutor` enable the use of all the
  computer cores available on a single computer. :class:`.BatchPoolExecutor`
  enables the use of an effectively arbitrary number of cloud computer cores.
  
  Functions provided to :meth:`.submit` are serialized using `dill
  <https://dill.readthedocs.io/en/latest/dill.html>`__, sent to a Python
  docker container in the cloud, deserialized, and executed. The results are
  serialized and returned to the machine from which :meth:`.submit` was
  called. The Python version in the docker container will share a major and
  minor verison with the local process. The `image` parameter overrides this
  behavior.
  
  When used as a context manager (the ``with`` syntax), the executor will wait
  for all jobs to finish before finishing the ``with`` statement. This
  behavior can be controlled by the `wait_on_exit` parameter.
  
  This class creates a folder ``batch-pool-executor`` at the root of the
  bucket specified by the `backend`. This folder can be safely deleted after
  all jobs have completed.
  
  Examples
  --------
  
  Add ``3`` to ``6`` on a machine in the cloud and send the result back to
  this machine:
  
  >>> with BatchPoolExecutor() as bpe:  # doctest: +SKIP
  ...     future_nine = bpe.submit(lambda: 3 + 6)
  >>> future_nine.result()  # doctest: +SKIP
  9
  
  :meth:`.map` facilitates the common case of executing a function on many
  values in parallel:
  
  >>> with BatchPoolExecutor() as bpe:  # doctest: +SKIP
  ...     list(bpe.map(lambda x: x * 3, range(4)))
  [0, 3, 6, 9]
  
  Parameters
  ----------
  name:
      A name for the executor. Executors produce many batches and each batch
      will include this name as a prefix.
  backend:
      Backend used to execute the jobs. Must be a :class:`.ServiceBackend`.
  image:
      The name of a Docker image used for each submitted job. The image must
      include Python 3.9 or later and must have the ``dill`` Python package
      installed. If you intend to use ``numpy``, ensure that OpenBLAS is also
      installed. If unspecified, an image with a matching Python verison and
      ``numpy``, ``scipy``, and ``sklearn`` installed is used.
  cpus_per_job:
      The number of CPU cores to allocate to each job. The default value is
      ``1``. The parameter is passed unaltered to :meth:`.Job.cpu`. This
      parameter's value is used to set several environment variables
      instructing BLAS and LAPACK to limit core use.
  wait_on_exit:
      If ``True`` or unspecified, wait for all jobs to complete when exiting a
      context. If ``False``, do not wait. This option has no effect if this
      executor is not used with the ``with`` syntax.
  cleanup_bucket:
      If ``True`` or unspecified, delete all temporary files in the cloud
      storage bucket when this executor fully shuts down. If Python crashes
      before the executor is shutdown, the files will not be deleted.
  project:
      DEPRECATED. Please specify gcs_requester_pays_configuration in :class:`.ServiceBackend`.
class_docstr_tok: ["An","executor","which","executes","Python","functions","in","the","cloud",".",":","class",":","`",".concurrent.futures.ProcessPoolExecutor","`","and",":","class",":","`",".concurrent.futures.ThreadPoolExecutor","`","enable","the","use","of","all","the","computer","cores","available","on","a","single","computer",".",":","class",":","`",".BatchPoolExecutor","`","enables","the","use","of","an","effectively","arbitrary","number","of","cloud","computer","cores",".","Functions","provided","to",":","meth",":","`",".submit","`","are","serialized","using","`","dill","<","https",":","\/\/dill.readthedocs.io\/en\/latest\/dill.html",">","`","__",",","sent","to","a","Python","docker","container","in","the","cloud",",","deserialized",",","and","executed",".","The","results","are","serialized","and","returned","to","the","machine","from","which",":","meth",":","`",".submit","`","was","called",".","The","Python","version","in","the","docker","container","will","share","a","major","and","minor","verison","with","the","local","process",".","The","`","image","`","parameter","overrides","this","behavior",".","When","used","as","a","context","manager","(","the","``","with","``","syntax",")",",","the","executor","will","wait","for","all","jobs","to","finish","before","finishing","the","``","with","``","statement",".","This","behavior","can","be","controlled","by","the","`","wait_on_exit","`","parameter",".","This","class","creates","a","folder","``","batch-pool-executor","``","at","the","root","of","the","bucket","specified","by","the","`","backend","`",".","This","folder","can","be","safely","deleted","after","all","jobs","have","completed",".","Examples","--","--","--","--","Add","``","3","``","to","``","6","``","on","a","machine","in","the","cloud","and","send","the","result","back","to","this","machine",":",">",">",">","with","BatchPoolExecutor","(",")","as","bpe",":","#","doctest",":","+SKIP","...","future_nine","=","bpe.submit","(","lambda",":","3","+","6",")",">",">",">","future_nine.result","(",")","#","doctest",":","+SKIP","9",":","meth",":","`",".map","`","facilitates","the","common","case","of","executing","a","function","on","many","values","in","parallel",":",">",">",">","with","BatchPoolExecutor","(",")","as","bpe",":","#","doctest",":","+SKIP","...","list","(","bpe.map","(","lambda","x",":","x","*","3",",","range","(","4",")",")",")","[","0",",","3",",","6",",","9","]","Parameters","--","--","--","--","--","name",":","A","name","for","the","executor",".","Executors","produce","many","batches","and","each","batch","will","include","this","name","as","a","prefix",".","backend",":","Backend","used","to","execute","the","jobs",".","Must","be","a",":","class",":","`",".ServiceBackend","`",".","image",":","The","name","of","a","Docker","image","used","for","each","submitted","job",".","The","image","must","include","Python","3.9","or","later","and","must","have","the","``","dill","``","Python","package","installed",".","If","you","intend","to","use","``","numpy","``",",","ensure","that","OpenBLAS","is","also","installed",".","If","unspecified",",","an","image","with","a","matching","Python","verison","and","``","numpy","``",",","``","scipy","``",",","and","``","sklearn","``","installed","is","used",".","cpus_per_job",":","The","number","of","CPU","cores","to","allocate","to","each","job",".","The","default","value","is","``","1","``",".","The","parameter","is","passed","unaltered","to",":","meth",":","`",".Job.cpu","`",".","This","parameter","'s","value","is","used","to","set","several","environment","variables","instructing","BLAS","and","LAPACK","to","limit","core","use",".","wait_on_exit",":","If","``","True","``","or","unspecified",",","wait","for","all","jobs","to","complete","when","exiting","a","context",".","If","``","False","``",",","do","not","wait",".","This","option","has","no","effect","if","this","executor","is","not","used","with","the","``","with","``","syntax",".","cleanup_bucket",":","If","``","True","``","or","unspecified",",","delete","all","temporary","files","in","the","cloud","storage","bucket","when","this","executor","fully","shuts","down",".","If","Python","crashes","before","the","executor","is","shutdown",",","the","files","will","not","be","deleted",".","project",":","DEPRECATED",".","Please","specify","gcs_requester_pays_configuration","in",":","class",":","`",".ServiceBackend","`","."]
docstr: Call `fn` on cloud machines with arguments from `iterables`.
  
  This function returns a generator which will produce each result in the
  same order as the `iterables`, only blocking if the result is not yet
  ready. You can convert the generator to a list with :class:`.list`.
  
  Examples
  --------
  
  Do nothing, but on the cloud:
  
  >>> with BatchPoolExecutor() as bpe:  # doctest: +SKIP
  ...     list(bpe.map(lambda x: x, range(4)))
  [0, 1, 2, 3]
  
  Call a function with two parameters, on the cloud:
  
  >>> with BatchPoolExecutor() as bpe:  # doctest: +SKIP
  ...     list(bpe.map(lambda x, y: x + y,
  ...                  ["white", "cat", "best"],
  ...                  ["house", "dog", "friend"]))
  ["whitehouse", "catdog", "bestfriend"]
  
  Generate products of random matrices, on the cloud:
  
  >>> def random_product(seed):
  ...     np.random.seed(seed)
  ...     w = np.random.rand(1, 100)
  ...     u = np.random.rand(100, 1)
  ...     return float(w @ u)
  >>> with BatchPoolExecutor() as bpe:  # doctest: +SKIP
  ...     list(bpe.map(random_product, range(4)))
  [24.440006386777277, 23.325755364428026, 23.920184804993806, 25.47912882125101]
  
  Parameters
  ----------
  fn:
      The function to execute.
  iterables:
      The `iterables` are zipped together and each tuple is used as
      arguments to `fn`. See the second example for more detail. It is not
      possible to pass keyword arguments. Each element of `iterables` must
      have the same length.
  timeout:
      This is roughly a timeout on how long we wait on each function
      call. Specifically, each call to the returned generator's
      :class:`.BatchPoolFuture`
      :meth:`.iterator.__next__` invokes :meth:`.BatchPoolFuture.result` with this
      `timeout`.
  chunksize:
      The number of tasks to schedule in the same docker container. Docker
      containers take about 5 seconds to start. Ideally, each task should
      take an order of magnitude more time than start-up time. You can
      make the chunksize larger to reduce parallelism but increase the
      amount of meaningful work done per-container.
docstr_tok: ["Call","`","fn","`","on","cloud","machines","with","arguments","from","`","iterables","`",".","This","function","returns","a","generator","which","will","produce","each","result","in","the","same","order","as","the","`","iterables","`",",","only","blocking","if","the","result","is","not","yet","ready",".","You","can","convert","the","generator","to","a","list","with",":","class",":","`",".list","`",".","Examples","--","--","--","--","Do","nothing",",","but","on","the","cloud",":",">",">",">","with","BatchPoolExecutor","(",")","as","bpe",":","#","doctest",":","+SKIP","...","list","(","bpe.map","(","lambda","x",":","x",",","range","(","4",")",")",")","[","0",",","1",",","2",",","3","]","Call","a","function","with","two","parameters",",","on","the","cloud",":",">",">",">","with","BatchPoolExecutor","(",")","as","bpe",":","#","doctest",":","+SKIP","...","list","(","bpe.map","(","lambda","x",",","y",":","x","+","y",",","...","[","``","white","''",",","``","cat","''",",","``","best","''","]",",","...","[","``","house","''",",","``","dog","''",",","``","friend","''","]",")",")","[","``","whitehouse","''",",","``","catdog","''",",","``","bestfriend","''","]","Generate","products","of","random","matrices",",","on","the","cloud",":",">",">",">","def","random_product","(","seed",")",":","...","np.random.seed","(","seed",")","...","w","=","np.random.rand","(","1",",","100",")","...","u","=","np.random.rand","(","100",",","1",")","...","return","float","(","w","@","u",")",">",">",">","with","BatchPoolExecutor","(",")","as","bpe",":","#","doctest",":","+SKIP","...","list","(","bpe.map","(","random_product",",","range","(","4",")",")",")","[","24.440006386777277",",","23.325755364428026",",","23.920184804993806",",","25.47912882125101","]","Parameters","--","--","--","--","--","fn",":","The","function","to","execute",".","iterables",":","The","`","iterables","`","are","zipped","together","and","each","tuple","is","used","as","arguments","to","`","fn","`",".","See","the","second","example","for","more","detail",".","It","is","not","possible","to","pass","keyword","arguments",".","Each","element","of","`","iterables","`","must","have","the","same","length",".","timeout",":","This","is","roughly","a","timeout","on","how","long","we","wait","on","each","function","call",".","Specifically",",","each","call","to","the","returned","generator's",":","class",":","`",".BatchPoolFuture","`",":","meth",":","`",".iterator.__next__","`","invokes",":","meth",":","`",".BatchPoolFuture.result","`","with","this","`","timeout","`",".","chunksize",":","The","number","of","tasks","to","schedule","in","the","same","docker","container",".","Docker","containers","take","about","5","seconds","to","start",".","Ideally",",","each","task","should","take","an","order","of","magnitude","more","time","than","start-up","time",".","You","can","make","the","chunksize","larger","to","reduce","parallelism","but","increase","the","amount","of","meaningful","work","done","per-container","."]
code: def map(
      self,
      fn: Callable,
      *iterables: Iterable[Any],
      timeout: Optional[Union[int, float]] = None,
      chunksize: int = 1,
  ):
      """Call `fn` on cloud machines with arguments from `iterables`.
  
      This function returns a generator which will produce each result in the
      same order as the `iterables`, only blocking if the result is not yet
      ready. You can convert the generator to a list with :class:`.list`.
  
      Examples
      --------
  
      Do nothing, but on the cloud:
  
      >>> with BatchPoolExecutor() as bpe:  # doctest: +SKIP
      ...     list(bpe.map(lambda x: x, range(4)))
      [0, 1, 2, 3]
  
      Call a function with two parameters, on the cloud:
  
      >>> with BatchPoolExecutor() as bpe:  # doctest: +SKIP
      ...     list(bpe.map(lambda x, y: x + y,
      ...                  ["white", "cat", "best"],
      ...                  ["house", "dog", "friend"]))
      ["whitehouse", "catdog", "bestfriend"]
  
      Generate products of random matrices, on the cloud:
  
      >>> def random_product(seed):
      ...     np.random.seed(seed)
      ...     w = np.random.rand(1, 100)
      ...     u = np.random.rand(100, 1)
      ...     return float(w @ u)
      >>> with BatchPoolExecutor() as bpe:  # doctest: +SKIP
      ...     list(bpe.map(random_product, range(4)))
      [24.440006386777277, 23.325755364428026, 23.920184804993806, 25.47912882125101]
  
      Parameters
      ----------
      fn:
          The function to execute.
      iterables:
          The `iterables` are zipped together and each tuple is used as
          arguments to `fn`. See the second example for more detail. It is not
          possible to pass keyword arguments. Each element of `iterables` must
          have the same length.
      timeout:
          This is roughly a timeout on how long we wait on each function
          call. Specifically, each call to the returned generator's
          :class:`.BatchPoolFuture`
          :meth:`.iterator.__next__` invokes :meth:`.BatchPoolFuture.result` with this
          `timeout`.
      chunksize:
          The number of tasks to schedule in the same docker container. Docker
          containers take about 5 seconds to start. Ideally, each task should
          take an order of magnitude more time than start-up time. You can
          make the chunksize larger to reduce parallelism but increase the
          amount of meaningful work done per-container.
      """
  
      agen = async_to_blocking(
          self.async_map(
              fn, iterables, timeout=timeout, chunksize=chunksize
          )
      )
  
      def generator_from_async_generator(aiter):
          try:
              while True:
                  yield async_to_blocking(aiter.__anext__())
          except StopAsyncIteration:
              return
  
      return generator_from_async_generator(agen.__aiter__())
  
imports: from typing import Optional, Callable, Type, Union, List, Any, Iterable, AsyncGenerator
  from types import TracebackType
  from io import BytesIO
  import warnings
  import asyncio
  import concurrent.futures
  import dill
  import functools
  from hailtop.utils import secret_alnum_string, partition, async_to_blocking
  import hailtop.batch_client.aioclient
  from hailtop.batch_client.parse import parse_cpu_in_mcpu
  from hailtop.aiotools.router_fs import RouterAsyncFS
  from .batch import Batch
  from .backend import ServiceBackend, HAIL_GENETICS_HAIL_IMAGE
label_desc: Use image node_id 4 for calling the BatchPoolExecutor obj's underlying member method code with example usage: obj.map(fn) and returns: generator_from_async_generator