class_docstr: Handles Amazon SageMaker processing tasks for jobs using Spark.
  
  Base class for either PySpark or SparkJars.
class_docstr_tok: ["Handles","Amazon","SageMaker","processing","tasks","for","jobs","using","Spark",".","Base","class","for","either","PySpark","or","SparkJars","."]
docstr: Initialize a ``_SparkProcessorBase`` instance.
  
  The _SparkProcessorBase handles Amazon SageMaker processing tasks for
  jobs using SageMaker Spark.
  
  Args:
      framework_version (str): The version of SageMaker PySpark.
      py_version (str): The version of python.
      container_version (str): The version of spark container.
      role (str): An AWS IAM role name or ARN. Amazon SageMaker Processing
          uses this role to access AWS resources, such as
          data stored in Amazon S3 (default: None).
          If not specified, the value from the defaults configuration file
          will be used.
      instance_type (str): Type of EC2 instance to use for
          processing, for example, 'ml.c4.xlarge'.
      instance_count (int): The number of instances to run
          the Processing job with. Defaults to 1.
      volume_size_in_gb (int): Size in GB of the EBS volume to
          use for storing data during processing (default: 30).
      volume_kms_key (str): A KMS key for the processing
          volume.
      output_kms_key (str): The KMS key id for all ProcessingOutputs.
      configuration_location (str): The S3 prefix URI where the user-provided EMR
          application configuration will be uploaded (default: None). If not specified,
          the default ``configuration location`` is 's3://{sagemaker-default-bucket}'.
      dependency_location (str): The S3 prefix URI where Spark dependencies will be
          uploaded (default: None). If not specified, the default ``dependency location``
          is 's3://{sagemaker-default-bucket}'.
      max_runtime_in_seconds (int): Timeout in seconds.
          After this amount of time Amazon SageMaker terminates the job
          regardless of its current status.
      base_job_name (str): Prefix for processing name. If not specified,
          the processor generates a default job name, based on the
          training image name and current timestamp.
      sagemaker_session (sagemaker.session.Session): Session object which
          manages interactions with Amazon
          SageMaker APIs and any other AWS services needed. If not specified,
          the processor creates one using the default AWS configuration chain.
      env (dict): Environment variables to be passed to the processing job.
      tags (Optional[Tags]): List of tags to be passed to the processing job.
      network_config (sagemaker.network.NetworkConfig): A NetworkConfig
          object that configures network isolation, encryption of
          inter-container traffic, security group IDs, and subnets.
docstr_tok: ["Initialize","a","``","_SparkProcessorBase","``","instance",".","The","_SparkProcessorBase","handles","Amazon","SageMaker","processing","tasks","for","jobs","using","SageMaker","Spark",".","Args",":","framework_version","(","str",")",":","The","version","of","SageMaker","PySpark",".","py_version","(","str",")",":","The","version","of","python",".","container_version","(","str",")",":","The","version","of","spark","container",".","role","(","str",")",":","An","AWS","IAM","role","name","or","ARN",".","Amazon","SageMaker","Processing","uses","this","role","to","access","AWS","resources",",","such","as","data","stored","in","Amazon","S3","(","default",":","None",")",".","If","not","specified",",","the","value","from","the","defaults","configuration","file","will","be","used",".","instance_type","(","str",")",":","Type","of","EC2","instance","to","use","for","processing",",","for","example",",","'ml.c4.xlarge","'",".","instance_count","(","int",")",":","The","number","of","instances","to","run","the","Processing","job","with",".","Defaults","to","1.","volume_size_in_gb","(","int",")",":","Size","in","GB","of","the","EBS","volume","to","use","for","storing","data","during","processing","(","default",":","30",")",".","volume_kms_key","(","str",")",":","A","KMS","key","for","the","processing","volume",".","output_kms_key","(","str",")",":","The","KMS","key","id","for","all","ProcessingOutputs",".","configuration_location","(","str",")",":","The","S3","prefix","URI","where","the","user-provided","EMR","application","configuration","will","be","uploaded","(","default",":","None",")",".","If","not","specified",",","the","default","``","configuration","location","``","is","'s3",":","\/\/","{","sagemaker-default-bucket","}","'",".","dependency_location","(","str",")",":","The","S3","prefix","URI","where","Spark","dependencies","will","be","uploaded","(","default",":","None",")",".","If","not","specified",",","the","default","``","dependency","location","``","is","'s3",":","\/\/","{","sagemaker-default-bucket","}","'",".","max_runtime_in_seconds","(","int",")",":","Timeout","in","seconds",".","After","this","amount","of","time","Amazon","SageMaker","terminates","the","job","regardless","of","its","current","status",".","base_job_name","(","str",")",":","Prefix","for","processing","name",".","If","not","specified",",","the","processor","generates","a","default","job","name",",","based","on","the","training","image","name","and","current","timestamp",".","sagemaker_session","(","sagemaker.session.Session",")",":","Session","object","which","manages","interactions","with","Amazon","SageMaker","APIs","and","any","other","AWS","services","needed",".","If","not","specified",",","the","processor","creates","one","using","the","default","AWS","configuration","chain",".","env","(","dict",")",":","Environment","variables","to","be","passed","to","the","processing","job",".","tags","(","Optional","[","Tags","]",")",":","List","of","tags","to","be","passed","to","the","processing","job",".","network_config","(","sagemaker.network.NetworkConfig",")",":","A","NetworkConfig","object","that","configures","network","isolation",",","encryption","of","inter-container","traffic",",","security","group","IDs",",","and","subnets","."]
code: def __init__(
      self,
      role=None,
      instance_type=None,
      instance_count=None,
      framework_version=None,
      py_version=None,
      container_version=None,
      image_uri=None,
      volume_size_in_gb=30,
      volume_kms_key=None,
      output_kms_key=None,
      configuration_location: Optional[str] = None,
      dependency_location: Optional[str] = None,
      max_runtime_in_seconds=None,
      base_job_name=None,
      sagemaker_session=None,
      env=None,
      tags=None,
      network_config=None,
  ):
      """Initialize a ``_SparkProcessorBase`` instance.
  
      The _SparkProcessorBase handles Amazon SageMaker processing tasks for
      jobs using SageMaker Spark.
  
      Args:
          framework_version (str): The version of SageMaker PySpark.
          py_version (str): The version of python.
          container_version (str): The version of spark container.
          role (str): An AWS IAM role name or ARN. Amazon SageMaker Processing
              uses this role to access AWS resources, such as
              data stored in Amazon S3 (default: None).
              If not specified, the value from the defaults configuration file
              will be used.
          instance_type (str): Type of EC2 instance to use for
              processing, for example, 'ml.c4.xlarge'.
          instance_count (int): The number of instances to run
              the Processing job with. Defaults to 1.
          volume_size_in_gb (int): Size in GB of the EBS volume to
              use for storing data during processing (default: 30).
          volume_kms_key (str): A KMS key for the processing
              volume.
          output_kms_key (str): The KMS key id for all ProcessingOutputs.
          configuration_location (str): The S3 prefix URI where the user-provided EMR
              application configuration will be uploaded (default: None). If not specified,
              the default ``configuration location`` is 's3://{sagemaker-default-bucket}'.
          dependency_location (str): The S3 prefix URI where Spark dependencies will be
              uploaded (default: None). If not specified, the default ``dependency location``
              is 's3://{sagemaker-default-bucket}'.
          max_runtime_in_seconds (int): Timeout in seconds.
              After this amount of time Amazon SageMaker terminates the job
              regardless of its current status.
          base_job_name (str): Prefix for processing name. If not specified,
              the processor generates a default job name, based on the
              training image name and current timestamp.
          sagemaker_session (sagemaker.session.Session): Session object which
              manages interactions with Amazon
              SageMaker APIs and any other AWS services needed. If not specified,
              the processor creates one using the default AWS configuration chain.
          env (dict): Environment variables to be passed to the processing job.
          tags (Optional[Tags]): List of tags to be passed to the processing job.
          network_config (sagemaker.network.NetworkConfig): A NetworkConfig
              object that configures network isolation, encryption of
              inter-container traffic, security group IDs, and subnets.
      """
      self.configuration_location = configuration_location
      self.dependency_location = dependency_location
      self.history_server = None
      self._spark_event_logs_s3_uri = None
  
      session = sagemaker_session or Session()
      region = session.boto_region_name
  
      self.image_uri = self._retrieve_image_uri(
          image_uri,
          framework_version,
          py_version,
          container_version,
          region,
          instance_type,
      )
  
      env = env or {}
      command = [_SparkProcessorBase._default_command]
  
      super(_SparkProcessorBase, self).__init__(
          role=role,
          image_uri=self.image_uri,
          instance_count=instance_count,
          instance_type=instance_type,
          command=command,
          volume_size_in_gb=volume_size_in_gb,
          volume_kms_key=volume_kms_key,
          output_kms_key=output_kms_key,
          max_runtime_in_seconds=max_runtime_in_seconds,
          base_job_name=base_job_name,
          sagemaker_session=session,
          env=env,
          tags=format_tags(tags),
          network_config=network_config,
      )
  
imports: from __future__ import absolute_import
  import json
  import logging
  import os.path
  import shutil
  import subprocess
  import tempfile
  import time
  import urllib.request
  from enum import Enum
  from io import BytesIO
  from urllib.parse import urlparse
  from copy import copy
  from typing import Union, List, Dict, Optional
  from sagemaker import image_uris, s3
  from sagemaker.local.image import _ecr_login_if_needed, _pull_image
  from sagemaker.processing import ProcessingInput, ProcessingOutput, ScriptProcessor
  from sagemaker.s3 import S3Uploader
  from sagemaker.session import Session
  from sagemaker.network import NetworkConfig
  from sagemaker.spark import defaults
  from sagemaker.utils import format_tags, Tags
  from sagemaker.workflow import is_pipeline_variable
  from sagemaker.workflow.pipeline_context import runnable_by_pipeline
  from sagemaker.workflow.entities import PipelineVariable
  from sagemaker.workflow.functions import Join
label_desc: Use image node_id 1 to create a new _SparkProcessorBase object from inherited base classes: ScriptProcessor with example: obj = _SparkProcessorBase(role, instance_type, instance_count, framework_version, py_version, container_version, image_uri, volume_size_in_gb, volume_kms_key, output_kms_key, configuration_location, dependency_location, max_runtime_in_seconds, base_job_name, sagemaker_session, env, tags, network_config)