class_docstr: Facilitate Featurization of In-memory objects.
  
  When featurizing a dataset, it's often the case that the initial set of
  data (pre-featurization) fits handily within memory. (For example, perhaps
  it fits within a column of a pandas DataFrame.) In this case, it would be
  convenient to directly be able to featurize this column of data. However,
  the process of featurization often generates large arrays which quickly eat
  up available memory. This class provides convenient capabilities to process
  such in-memory data by checkpointing generated features periodically to
  disk.
  
  Example
  -------
  Here's an example with only datapoints and no labels or weights.
  
  >>> import deepchem as dc
  >>> smiles = ["C", "CC", "CCC", "CCCC"]
  >>> featurizer = dc.feat.CircularFingerprint()
  >>> loader = dc.data.InMemoryLoader(tasks=["task1"], featurizer=featurizer)
  >>> dataset = loader.create_dataset(smiles, shard_size=2)
  >>> len(dataset)
  4
  
  Here's an example with both datapoints and labels
  
  >>> import deepchem as dc
  >>> smiles = ["C", "CC", "CCC", "CCCC"]
  >>> labels = [1, 0, 1, 0]
  >>> featurizer = dc.feat.CircularFingerprint()
  >>> loader = dc.data.InMemoryLoader(tasks=["task1"], featurizer=featurizer)
  >>> dataset = loader.create_dataset(zip(smiles, labels), shard_size=2)
  >>> len(dataset)
  4
  
  Here's an example with datapoints, labels, weights and ids all provided.
  
  >>> import deepchem as dc
  >>> smiles = ["C", "CC", "CCC", "CCCC"]
  >>> labels = [1, 0, 1, 0]
  >>> weights = [1.5, 0, 1.5, 0]
  >>> ids = ["C", "CC", "CCC", "CCCC"]
  >>> featurizer = dc.feat.CircularFingerprint()
  >>> loader = dc.data.InMemoryLoader(tasks=["task1"], featurizer=featurizer)
  >>> dataset = loader.create_dataset(zip(smiles, labels, weights, ids), shard_size=2)
  >>> len(dataset)
  4
class_docstr_tok: ["Facilitate","Featurization","of","In-memory","objects",".","When","featurizing","a","dataset",",","it","'s","often","the","case","that","the","initial","set","of","data","(","pre-featurization",")","fits","handily","within","memory",".","(","For","example",",","perhaps","it","fits","within","a","column","of","a","pandas","DataFrame",".",")","In","this","case",",","it","would","be","convenient","to","directly","be","able","to","featurize","this","column","of","data",".","However",",","the","process","of","featurization","often","generates","large","arrays","which","quickly","eat","up","available","memory",".","This","class","provides","convenient","capabilities","to","process","such","in-memory","data","by","checkpointing","generated","features","periodically","to","disk",".","Example","--","--","--","-","Here","'s","an","example","with","only","datapoints","and","no","labels","or","weights",".",">",">",">","import","deepchem","as","dc",">",">",">","smiles","=","[","``","C","''",",","``","CC","''",",","``","CCC","''",",","``","CCCC","''","]",">",">",">","featurizer","=","dc.feat.CircularFingerprint","(",")",">",">",">","loader","=","dc.data.InMemoryLoader","(","tasks=","[","``","task1","''","]",",","featurizer=featurizer",")",">",">",">","dataset","=","loader.create_dataset","(","smiles",",","shard_size=2",")",">",">",">","len","(","dataset",")","4","Here","'s","an","example","with","both","datapoints","and","labels",">",">",">","import","deepchem","as","dc",">",">",">","smiles","=","[","``","C","''",",","``","CC","''",",","``","CCC","''",",","``","CCCC","''","]",">",">",">","labels","=","[","1",",","0",",","1",",","0","]",">",">",">","featurizer","=","dc.feat.CircularFingerprint","(",")",">",">",">","loader","=","dc.data.InMemoryLoader","(","tasks=","[","``","task1","''","]",",","featurizer=featurizer",")",">",">",">","dataset","=","loader.create_dataset","(","zip","(","smiles",",","labels",")",",","shard_size=2",")",">",">",">","len","(","dataset",")","4","Here","'s","an","example","with","datapoints",",","labels",",","weights","and","ids","all","provided",".",">",">",">","import","deepchem","as","dc",">",">",">","smiles","=","[","``","C","''",",","``","CC","''",",","``","CCC","''",",","``","CCCC","''","]",">",">",">","labels","=","[","1",",","0",",","1",",","0","]",">",">",">","weights","=","[","1.5",",","0",",","1.5",",","0","]",">",">",">","ids","=","[","``","C","''",",","``","CC","''",",","``","CCC","''",",","``","CCCC","''","]",">",">",">","featurizer","=","dc.feat.CircularFingerprint","(",")",">",">",">","loader","=","dc.data.InMemoryLoader","(","tasks=","[","``","task1","''","]",",","featurizer=featurizer",")",">",">",">","dataset","=","loader.create_dataset","(","zip","(","smiles",",","labels",",","weights",",","ids",")",",","shard_size=2",")",">",">",">","len","(","dataset",")","4"]
docstr: Creates and returns a `Dataset` object by featurizing provided files.
  
  Reads in `inputs` and uses `self.featurizer` to featurize the
  data in these input files.  For large files, automatically shards
  into smaller chunks of `shard_size` datapoints for convenience.
  Returns a `Dataset` object that contains the featurized dataset.
  
  This implementation assumes that the helper methods `_get_shards`
  and `_featurize_shard` are implemented and that each shard
  returned by `_get_shards` is a pandas dataframe.  You may choose
  to reuse or override this method in your subclass implementations.
  
  Parameters
  ----------
  inputs: Sequence[Any]
      List of inputs to process. Entries can be arbitrary objects so long as
      they are understood by `self.featurizer`
  data_dir: str, optional (default None)
      Directory to store featurized dataset.
  shard_size: int, optional (default 8192)
      Number of examples stored in each shard.
  
  Returns
  -------
  DiskDataset
      A `DiskDataset` object containing a featurized representation of data
      from `inputs`.
docstr_tok: ["Creates","and","returns","a","`","Dataset","`","object","by","featurizing","provided","files",".","Reads","in","`","inputs","`","and","uses","`","self.featurizer","`","to","featurize","the","data","in","these","input","files",".","For","large","files",",","automatically","shards","into","smaller","chunks","of","`","shard_size","`","datapoints","for","convenience",".","Returns","a","`","Dataset","`","object","that","contains","the","featurized","dataset",".","This","implementation","assumes","that","the","helper","methods","`","_get_shards","`","and","`","_featurize_shard","`","are","implemented","and","that","each","shard","returned","by","`","_get_shards","`","is","a","pandas","dataframe",".","You","may","choose","to","reuse","or","override","this","method","in","your","subclass","implementations",".","Parameters","--","--","--","--","--","inputs",":","Sequence","[","Any","]","List","of","inputs","to","process",".","Entries","can","be","arbitrary","objects","so","long","as","they","are","understood","by","`","self.featurizer","`","data_dir",":","str",",","optional","(","default","None",")","Directory","to","store","featurized","dataset",".","shard_size",":","int",",","optional","(","default","8192",")","Number","of","examples","stored","in","each","shard",".","Returns","--","--","--","-","DiskDataset","A","`","DiskDataset","`","object","containing","a","featurized","representation","of","data","from","`","inputs","`","."]
code: def create_dataset(
      self,
      inputs: Sequence[Any],
      data_dir: Optional[str] = None,
      shard_size: Optional[int] = 8192,
  ) -> DiskDataset:
      """Creates and returns a `Dataset` object by featurizing provided files.
  
      Reads in `inputs` and uses `self.featurizer` to featurize the
      data in these input files.  For large files, automatically shards
      into smaller chunks of `shard_size` datapoints for convenience.
      Returns a `Dataset` object that contains the featurized dataset.
  
      This implementation assumes that the helper methods `_get_shards`
      and `_featurize_shard` are implemented and that each shard
      returned by `_get_shards` is a pandas dataframe.  You may choose
      to reuse or override this method in your subclass implementations.
  
      Parameters
      ----------
      inputs: Sequence[Any]
          List of inputs to process. Entries can be arbitrary objects so long as
          they are understood by `self.featurizer`
      data_dir: str, optional (default None)
          Directory to store featurized dataset.
      shard_size: int, optional (default 8192)
          Number of examples stored in each shard.
  
      Returns
      -------
      DiskDataset
          A `DiskDataset` object containing a featurized representation of data
          from `inputs`.
      """
      logger.info("Loading raw samples now.")
      logger.info("shard_size: %s" % str(shard_size))
  
      if not isinstance(inputs, list):
          try:
              inputs = list(inputs)
          except TypeError:
              inputs = [inputs]
  
      def shard_generator():
          global_index = 0
          for shard_num, shard in enumerate(
              self._get_shards(inputs, shard_size)
          ):
              time1 = time.time()
              X, y, w, ids = self._featurize_shard(shard, global_index)
              global_index += len(shard)
  
              time2 = time.time()
              logger.info(
                  "TIMING: featurizing shard %d took %0.3f s"
                  % (shard_num, time2 - time1)
              )
              yield X, y, w, ids
  
      return DiskDataset.create_dataset(
          shard_generator(), data_dir, self.tasks
      )
  
imports: import os
  import tempfile
  import zipfile
  import time
  import logging
  import warnings
  from typing import List, Optional, Tuple, Any, Sequence, Union, Iterator
  import pandas
  import numpy
  from deepchem.utils.typing import OneOrMany
  from deepchem.utils.data_utils import load_image_files, load_csv_files, load_json_files, load_sdf_files, unzip_file
  from deepchem.feat import UserDefinedFeaturizer, Featurizer
  from deepchem.data import Dataset, DiskDataset, NumpyDataset, ImageDataset
  from deepchem.feat.molecule_featurizers import OneHotFeaturizer
  from deepchem.utils.genomics_utils import encode_bio_sequence
label_desc: Use image node_id 1 for calling the InMemoryLoader obj's underlying member method code with example usage: obj.create_dataset(inputs, data_dir, shard_size) and returns: DiskDataset