class_docstr: The internal immutable DataFrame which manages Spark DataFrame and column names and index
  information.
  
  .. note:: this is an internal class. It is not supposed to be exposed to users and users
      should not directly access to it.
  
  The internal immutable DataFrame represents the index information for a DataFrame it belongs to.
  For instance, if we have a pandas-on-Spark DataFrame as below, pandas DataFrame does not
  store the index as columns.
  
  >>> psdf = ps.DataFrame({
  ...     'A': [1, 2, 3, 4],
  ...     'B': [5, 6, 7, 8],
  ...     'C': [9, 10, 11, 12],
  ...     'D': [13, 14, 15, 16],
  ...     'E': [17, 18, 19, 20]}, columns = ['A', 'B', 'C', 'D', 'E'])
  >>> psdf  # doctest: +NORMALIZE_WHITESPACE
     A  B   C   D   E
  0  1  5   9  13  17
  1  2  6  10  14  18
  2  3  7  11  15  19
  3  4  8  12  16  20
  
  However, all columns including index column are also stored in Spark DataFrame internally
  as below.
  
  >>> psdf._internal.to_internal_spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE
  +-----------------+---+---+---+---+---+
  |__index_level_0__|  A|  B|  C|  D|  E|
  +-----------------+---+---+---+---+---+
  |                0|  1|  5|  9| 13| 17|
  |                1|  2|  6| 10| 14| 18|
  |                2|  3|  7| 11| 15| 19|
  |                3|  4|  8| 12| 16| 20|
  +-----------------+---+---+---+---+---+
  
  To fill this gap, the current metadata is used by mapping Spark's internal column
  to pandas-on-Spark's index. See the method below:
  
  * `spark_frame` represents the internal Spark DataFrame
  
  * `data_spark_column_names` represents non-indexing Spark column names
  
  * `data_spark_columns` represents non-indexing Spark columns
  
  * `data_fields` represents non-indexing InternalFields
  
  * `index_spark_column_names` represents internal index Spark column names
  
  * `index_spark_columns` represents internal index Spark columns
  
  * `index_fields` represents index InternalFields
  
  * `spark_column_names` represents all columns
  
  * `index_names` represents the external index name as a label
  
  * `to_internal_spark_frame` represents Spark DataFrame derived by the metadata. Includes index.
  
  * `to_pandas_frame` represents pandas DataFrame derived by the metadata
  
  >>> internal = psdf._internal
  >>> internal.spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
  +-----------------+---+---+---+---+---+-----------------+
  |__index_level_0__|  A|  B|  C|  D|  E|__natural_order__|
  +-----------------+---+---+---+---+---+-----------------+
  |                0|  1|  5|  9| 13| 17|              ...|
  |                1|  2|  6| 10| 14| 18|              ...|
  |                2|  3|  7| 11| 15| 19|              ...|
  |                3|  4|  8| 12| 16| 20|              ...|
  +-----------------+---+---+---+---+---+-----------------+
  >>> internal.data_spark_column_names
  ['A', 'B', 'C', 'D', 'E']
  >>> internal.index_spark_column_names
  ['__index_level_0__']
  >>> internal.spark_column_names
  ['__index_level_0__', 'A', 'B', 'C', 'D', 'E']
  >>> internal.index_names
  [None]
  >>> internal.data_fields    # doctest: +NORMALIZE_WHITESPACE
  [InternalField(dtype=int64, struct_field=StructField('A', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('B', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('C', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('D', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('E', LongType(), False))]
  >>> internal.index_fields
  [InternalField(dtype=int64, struct_field=StructField('__index_level_0__', LongType(), False))]
  >>> internal.to_internal_spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE
  +-----------------+---+---+---+---+---+
  |__index_level_0__|  A|  B|  C|  D|  E|
  +-----------------+---+---+---+---+---+
  |                0|  1|  5|  9| 13| 17|
  |                1|  2|  6| 10| 14| 18|
  |                2|  3|  7| 11| 15| 19|
  |                3|  4|  8| 12| 16| 20|
  +-----------------+---+---+---+---+---+
  >>> internal.to_pandas_frame
     A  B   C   D   E
  0  1  5   9  13  17
  1  2  6  10  14  18
  2  3  7  11  15  19
  3  4  8  12  16  20
  
  In case that index is set to one of the existing columns as below:
  
  >>> psdf1 = psdf.set_index("A")
  >>> psdf1  # doctest: +NORMALIZE_WHITESPACE
     B   C   D   E
  A
  1  5   9  13  17
  2  6  10  14  18
  3  7  11  15  19
  4  8  12  16  20
  
  >>> psdf1._internal.to_internal_spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE
  +---+---+---+---+---+
  |  A|  B|  C|  D|  E|
  +---+---+---+---+---+
  |  1|  5|  9| 13| 17|
  |  2|  6| 10| 14| 18|
  |  3|  7| 11| 15| 19|
  |  4|  8| 12| 16| 20|
  +---+---+---+---+---+
  
  >>> internal = psdf1._internal
  >>> internal.spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
  +-----------------+---+---+---+---+---+-----------------+
  |__index_level_0__|  A|  B|  C|  D|  E|__natural_order__|
  +-----------------+---+---+---+---+---+-----------------+
  |                0|  1|  5|  9| 13| 17|              ...|
  |                1|  2|  6| 10| 14| 18|              ...|
  |                2|  3|  7| 11| 15| 19|              ...|
  |                3|  4|  8| 12| 16| 20|              ...|
  +-----------------+---+---+---+---+---+-----------------+
  >>> internal.data_spark_column_names
  ['B', 'C', 'D', 'E']
  >>> internal.index_spark_column_names
  ['A']
  >>> internal.spark_column_names
  ['A', 'B', 'C', 'D', 'E']
  >>> internal.index_names
  [('A',)]
  >>> internal.data_fields  # doctest: +NORMALIZE_WHITESPACE
  [InternalField(dtype=int64, struct_field=StructField('B', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('C', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('D', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('E', LongType(), False))]
  >>> internal.index_fields
  [InternalField(dtype=int64, struct_field=StructField('A', LongType(), False))]
  >>> internal.to_internal_spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE
  +---+---+---+---+---+
  |  A|  B|  C|  D|  E|
  +---+---+---+---+---+
  |  1|  5|  9| 13| 17|
  |  2|  6| 10| 14| 18|
  |  3|  7| 11| 15| 19|
  |  4|  8| 12| 16| 20|
  +---+---+---+---+---+
  >>> internal.to_pandas_frame  # doctest: +NORMALIZE_WHITESPACE
     B   C   D   E
  A
  1  5   9  13  17
  2  6  10  14  18
  3  7  11  15  19
  4  8  12  16  20
  
  In case that index becomes a multi index as below:
  
  >>> psdf2 = psdf.set_index("A", append=True)
  >>> psdf2  # doctest: +NORMALIZE_WHITESPACE
       B   C   D   E
    A
  0 1  5   9  13  17
  1 2  6  10  14  18
  2 3  7  11  15  19
  3 4  8  12  16  20
  
  >>> psdf2._internal.to_internal_spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE
  +-----------------+---+---+---+---+---+
  |__index_level_0__|  A|  B|  C|  D|  E|
  +-----------------+---+---+---+---+---+
  |                0|  1|  5|  9| 13| 17|
  |                1|  2|  6| 10| 14| 18|
  |                2|  3|  7| 11| 15| 19|
  |                3|  4|  8| 12| 16| 20|
  +-----------------+---+---+---+---+---+
  
  >>> internal = psdf2._internal
  >>> internal.spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
  +-----------------+---+---+---+---+---+-----------------+
  |__index_level_0__|  A|  B|  C|  D|  E|__natural_order__|
  +-----------------+---+---+---+---+---+-----------------+
  |                0|  1|  5|  9| 13| 17|              ...|
  |                1|  2|  6| 10| 14| 18|              ...|
  |                2|  3|  7| 11| 15| 19|              ...|
  |                3|  4|  8| 12| 16| 20|              ...|
  +-----------------+---+---+---+---+---+-----------------+
  >>> internal.data_spark_column_names
  ['B', 'C', 'D', 'E']
  >>> internal.index_spark_column_names
  ['__index_level_0__', 'A']
  >>> internal.spark_column_names
  ['__index_level_0__', 'A', 'B', 'C', 'D', 'E']
  >>> internal.index_names
  [None, ('A',)]
  >>> internal.data_fields  # doctest: +NORMALIZE_WHITESPACE
  [InternalField(dtype=int64, struct_field=StructField('B', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('C', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('D', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('E', LongType(), False))]
  >>> internal.index_fields  # doctest: +NORMALIZE_WHITESPACE
  [InternalField(dtype=int64, struct_field=StructField('__index_level_0__', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('A', LongType(), False))]
  >>> internal.to_internal_spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE
  +-----------------+---+---+---+---+---+
  |__index_level_0__|  A|  B|  C|  D|  E|
  +-----------------+---+---+---+---+---+
  |                0|  1|  5|  9| 13| 17|
  |                1|  2|  6| 10| 14| 18|
  |                2|  3|  7| 11| 15| 19|
  |                3|  4|  8| 12| 16| 20|
  +-----------------+---+---+---+---+---+
  >>> internal.to_pandas_frame  # doctest: +NORMALIZE_WHITESPACE
       B   C   D   E
    A
  0 1  5   9  13  17
  1 2  6  10  14  18
  2 3  7  11  15  19
  3 4  8  12  16  20
  
  For multi-level columns, it also holds column_labels
  
  >>> columns = pd.MultiIndex.from_tuples([('X', 'A'), ('X', 'B'),
  ...                                      ('Y', 'C'), ('Y', 'D')])
  >>> psdf3 = ps.DataFrame([
  ...     [1, 2, 3, 4],
  ...     [5, 6, 7, 8],
  ...     [9, 10, 11, 12],
  ...     [13, 14, 15, 16],
  ...     [17, 18, 19, 20]], columns = columns)
  >>> psdf3  # doctest: +NORMALIZE_WHITESPACE
      X       Y
      A   B   C   D
  0   1   2   3   4
  1   5   6   7   8
  2   9  10  11  12
  3  13  14  15  16
  4  17  18  19  20
  
  >>> internal = psdf3._internal
  >>> internal.spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
  +-----------------+------+------+------+------+-----------------+
  |__index_level_0__|(X, A)|(X, B)|(Y, C)|(Y, D)|__natural_order__|
  +-----------------+------+------+------+------+-----------------+
  |                0|     1|     2|     3|     4|              ...|
  |                1|     5|     6|     7|     8|              ...|
  |                2|     9|    10|    11|    12|              ...|
  |                3|    13|    14|    15|    16|              ...|
  |                4|    17|    18|    19|    20|              ...|
  +-----------------+------+------+------+------+-----------------+
  >>> internal.data_spark_column_names
  ['(X, A)', '(X, B)', '(Y, C)', '(Y, D)']
  >>> internal.column_labels
  [('X', 'A'), ('X', 'B'), ('Y', 'C'), ('Y', 'D')]
  
  For Series, it also holds scol to represent the column.
  
  >>> psseries = psdf1.B
  >>> psseries
  A
  1    5
  2    6
  3    7
  4    8
  Name: B, dtype: int64
  
  >>> internal = psseries._internal
  >>> internal.spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
  +-----------------+---+---+---+---+---+-----------------+
  |__index_level_0__|  A|  B|  C|  D|  E|__natural_order__|
  +-----------------+---+---+---+---+---+-----------------+
  |                0|  1|  5|  9| 13| 17|              ...|
  |                1|  2|  6| 10| 14| 18|              ...|
  |                2|  3|  7| 11| 15| 19|              ...|
  |                3|  4|  8| 12| 16| 20|              ...|
  +-----------------+---+---+---+---+---+-----------------+
  >>> internal.data_spark_column_names
  ['B']
  >>> internal.index_spark_column_names
  ['A']
  >>> internal.spark_column_names
  ['A', 'B']
  >>> internal.index_names
  [('A',)]
  >>> internal.data_fields
  [InternalField(dtype=int64, struct_field=StructField('B', LongType(), False))]
  >>> internal.index_fields
  [InternalField(dtype=int64, struct_field=StructField('A', LongType(), False))]
  >>> internal.to_internal_spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE
  +---+---+
  |  A|  B|
  +---+---+
  |  1|  5|
  |  2|  6|
  |  3|  7|
  |  4|  8|
  +---+---+
  >>> internal.to_pandas_frame  # doctest: +NORMALIZE_WHITESPACE
     B
  A
  1  5
  2  6
  3  7
  4  8
class_docstr_tok: ["The","internal","immutable","DataFrame","which","manages","Spark","DataFrame","and","column","names","and","index","information",".","..","note",":",":","this","is","an","internal","class",".","It","is","not","supposed","to","be","exposed","to","users","and","users","should","not","directly","access","to","it",".","The","internal","immutable","DataFrame","represents","the","index","information","for","a","DataFrame","it","belongs","to",".","For","instance",",","if","we","have","a","pandas-on-Spark","DataFrame","as","below",",","pandas","DataFrame","does","not","store","the","index","as","columns",".",">",">",">","psdf","=","ps.DataFrame","(","{","...","'","A","'",":","[","1",",","2",",","3",",","4","]",",","...","'","B","'",":","[","5",",","6",",","7",",","8","]",",","...","'","C","'",":","[","9",",","10",",","11",",","12","]",",","...","'D","'",":","[","13",",","14",",","15",",","16","]",",","...","'","E","'",":","[","17",",","18",",","19",",","20","]","}",",","columns","=","[","'","A","'",",","'","B","'",",","'","C","'",",","'D","'",",","'","E","'","]",")",">",">",">","psdf","#","doctest",":","+NORMALIZE_WHITESPACE","A","B","C","D","E","0","1","5","9","13","17","1","2","6","10","14","18","2","3","7","11","15","19","3","4","8","12","16","20","However",",","all","columns","including","index","column","are","also","stored","in","Spark","DataFrame","internally","as","below",".",">",">",">","psdf._internal.to_internal_spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","|__index_level_0__|","A|","B|","C|","D|","E|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","|","0|","1|","5|","9|","13|","17|","|","1|","2|","6|","10|","14|","18|","|","2|","3|","7|","11|","15|","19|","|","3|","4|","8|","12|","16|","20|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","To","fill","this","gap",",","the","current","metadata","is","used","by","mapping","Spark","'s","internal","column","to","pandas-on-Spark","'s","index",".","See","the","method","below",":","*","`","spark_frame","`","represents","the","internal","Spark","DataFrame","*","`","data_spark_column_names","`","represents","non-indexing","Spark","column","names","*","`","data_spark_columns","`","represents","non-indexing","Spark","columns","*","`","data_fields","`","represents","non-indexing","InternalFields","*","`","index_spark_column_names","`","represents","internal","index","Spark","column","names","*","`","index_spark_columns","`","represents","internal","index","Spark","columns","*","`","index_fields","`","represents","index","InternalFields","*","`","spark_column_names","`","represents","all","columns","*","`","index_names","`","represents","the","external","index","name","as","a","label","*","`","to_internal_spark_frame","`","represents","Spark","DataFrame","derived","by","the","metadata",".","Includes","index",".","*","`","to_pandas_frame","`","represents","pandas","DataFrame","derived","by","the","metadata",">",">",">","internal","=","psdf._internal",">",">",">","internal.spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+ELLIPSIS","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+","|__index_level_0__|","A|","B|","C|","D|","E|__natural_order__|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+","|","0|","1|","5|","9|","13|","17|","...","|","|","1|","2|","6|","10|","14|","18|","...","|","|","2|","3|","7|","11|","15|","19|","...","|","|","3|","4|","8|","12|","16|","20|","...","|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+",">",">",">","internal.data_spark_column_names","[","'","A","'",",","'","B","'",",","'","C","'",",","'D","'",",","'","E","'","]",">",">",">","internal.index_spark_column_names","[","'__index_level_0__","'","]",">",">",">","internal.spark_column_names","[","'__index_level_0__","'",",","'","A","'",",","'","B","'",",","'","C","'",",","'D","'",",","'","E","'","]",">",">",">","internal.index_names","[","None","]",">",">",">","internal.data_fields","#","doctest",":","+NORMALIZE_WHITESPACE","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","A","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","B","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","C","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'D","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","E","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.index_fields","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'__index_level_0__","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.to_internal_spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","|__index_level_0__|","A|","B|","C|","D|","E|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","|","0|","1|","5|","9|","13|","17|","|","1|","2|","6|","10|","14|","18|","|","2|","3|","7|","11|","15|","19|","|","3|","4|","8|","12|","16|","20|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+",">",">",">","internal.to_pandas_frame","A","B","C","D","E","0","1","5","9","13","17","1","2","6","10","14","18","2","3","7","11","15","19","3","4","8","12","16","20","In","case","that","index","is","set","to","one","of","the","existing","columns","as","below",":",">",">",">","psdf1","=","psdf.set_index","(","``","A","''",")",">",">",">","psdf1","#","doctest",":","+NORMALIZE_WHITESPACE","B","C","D","E","A","1","5","9","13","17","2","6","10","14","18","3","7","11","15","19","4","8","12","16","20",">",">",">","psdf1._internal.to_internal_spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+","--","-+","--","-+","--","-+","--","-+","--","-+","|","A|","B|","C|","D|","E|","+","--","-+","--","-+","--","-+","--","-+","--","-+","|","1|","5|","9|","13|","17|","|","2|","6|","10|","14|","18|","|","3|","7|","11|","15|","19|","|","4|","8|","12|","16|","20|","+","--","-+","--","-+","--","-+","--","-+","--","-+",">",">",">","internal","=","psdf1._internal",">",">",">","internal.spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+ELLIPSIS","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+","|__index_level_0__|","A|","B|","C|","D|","E|__natural_order__|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+","|","0|","1|","5|","9|","13|","17|","...","|","|","1|","2|","6|","10|","14|","18|","...","|","|","2|","3|","7|","11|","15|","19|","...","|","|","3|","4|","8|","12|","16|","20|","...","|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+",">",">",">","internal.data_spark_column_names","[","'","B","'",",","'","C","'",",","'D","'",",","'","E","'","]",">",">",">","internal.index_spark_column_names","[","'","A","'","]",">",">",">","internal.spark_column_names","[","'","A","'",",","'","B","'",",","'","C","'",",","'D","'",",","'","E","'","]",">",">",">","internal.index_names","[","(","'","A","'",",",")","]",">",">",">","internal.data_fields","#","doctest",":","+NORMALIZE_WHITESPACE","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","B","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","C","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'D","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","E","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.index_fields","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","A","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.to_internal_spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+","--","-+","--","-+","--","-+","--","-+","--","-+","|","A|","B|","C|","D|","E|","+","--","-+","--","-+","--","-+","--","-+","--","-+","|","1|","5|","9|","13|","17|","|","2|","6|","10|","14|","18|","|","3|","7|","11|","15|","19|","|","4|","8|","12|","16|","20|","+","--","-+","--","-+","--","-+","--","-+","--","-+",">",">",">","internal.to_pandas_frame","#","doctest",":","+NORMALIZE_WHITESPACE","B","C","D","E","A","1","5","9","13","17","2","6","10","14","18","3","7","11","15","19","4","8","12","16","20","In","case","that","index","becomes","a","multi","index","as","below",":",">",">",">","psdf2","=","psdf.set_index","(","``","A","''",",","append=True",")",">",">",">","psdf2","#","doctest",":","+NORMALIZE_WHITESPACE","B","C","D","E","A","0","1","5","9","13","17","1","2","6","10","14","18","2","3","7","11","15","19","3","4","8","12","16","20",">",">",">","psdf2._internal.to_internal_spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","|__index_level_0__|","A|","B|","C|","D|","E|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","|","0|","1|","5|","9|","13|","17|","|","1|","2|","6|","10|","14|","18|","|","2|","3|","7|","11|","15|","19|","|","3|","4|","8|","12|","16|","20|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+",">",">",">","internal","=","psdf2._internal",">",">",">","internal.spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+ELLIPSIS","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+","|__index_level_0__|","A|","B|","C|","D|","E|__natural_order__|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+","|","0|","1|","5|","9|","13|","17|","...","|","|","1|","2|","6|","10|","14|","18|","...","|","|","2|","3|","7|","11|","15|","19|","...","|","|","3|","4|","8|","12|","16|","20|","...","|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+",">",">",">","internal.data_spark_column_names","[","'","B","'",",","'","C","'",",","'D","'",",","'","E","'","]",">",">",">","internal.index_spark_column_names","[","'__index_level_0__","'",",","'","A","'","]",">",">",">","internal.spark_column_names","[","'__index_level_0__","'",",","'","A","'",",","'","B","'",",","'","C","'",",","'D","'",",","'","E","'","]",">",">",">","internal.index_names","[","None",",","(","'","A","'",",",")","]",">",">",">","internal.data_fields","#","doctest",":","+NORMALIZE_WHITESPACE","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","B","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","C","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'D","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","E","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.index_fields","#","doctest",":","+NORMALIZE_WHITESPACE","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'__index_level_0__","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","A","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.to_internal_spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","|__index_level_0__|","A|","B|","C|","D|","E|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","|","0|","1|","5|","9|","13|","17|","|","1|","2|","6|","10|","14|","18|","|","2|","3|","7|","11|","15|","19|","|","3|","4|","8|","12|","16|","20|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+",">",">",">","internal.to_pandas_frame","#","doctest",":","+NORMALIZE_WHITESPACE","B","C","D","E","A","0","1","5","9","13","17","1","2","6","10","14","18","2","3","7","11","15","19","3","4","8","12","16","20","For","multi-level","columns",",","it","also","holds","column_labels",">",">",">","columns","=","pd.MultiIndex.from_tuples","(","[","(","'","X","'",",","'","A","'",")",",","(","'","X","'",",","'","B","'",")",",","...","(","'","Y","'",",","'","C","'",")",",","(","'","Y","'",",","'D","'",")","]",")",">",">",">","psdf3","=","ps.DataFrame","(","[","...","[","1",",","2",",","3",",","4","]",",","...","[","5",",","6",",","7",",","8","]",",","...","[","9",",","10",",","11",",","12","]",",","...","[","13",",","14",",","15",",","16","]",",","...","[","17",",","18",",","19",",","20","]","]",",","columns","=","columns",")",">",">",">","psdf3","#","doctest",":","+NORMALIZE_WHITESPACE","X","Y","A","B","C","D","0","1","2","3","4","1","5","6","7","8","2","9","10","11","12","3","13","14","15","16","4","17","18","19","20",">",">",">","internal","=","psdf3._internal",">",">",">","internal.spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+ELLIPSIS","+","--","--","--","--","--","--","--","--","-+","--","--","--","+","--","--","--","+","--","--","--","+","--","--","--","+","--","--","--","--","--","--","--","--","-+","|__index_level_0__|","(","X",",","A",")","|","(","X",",","B",")","|","(","Y",",","C",")","|","(","Y",",","D",")","|__natural_order__|","+","--","--","--","--","--","--","--","--","-+","--","--","--","+","--","--","--","+","--","--","--","+","--","--","--","+","--","--","--","--","--","--","--","--","-+","|","0|","1|","2|","3|","4|","...","|","|","1|","5|","6|","7|","8|","...","|","|","2|","9|","10|","11|","12|","...","|","|","3|","13|","14|","15|","16|","...","|","|","4|","17|","18|","19|","20|","...","|","+","--","--","--","--","--","--","--","--","-+","--","--","--","+","--","--","--","+","--","--","--","+","--","--","--","+","--","--","--","--","--","--","--","--","-+",">",">",">","internal.data_spark_column_names","[","'","(","X",",","A",")","'",",","'","(","X",",","B",")","'",",","'","(","Y",",","C",")","'",",","'","(","Y",",","D",")","'","]",">",">",">","internal.column_labels","[","(","'","X","'",",","'","A","'",")",",","(","'","X","'",",","'","B","'",")",",","(","'","Y","'",",","'","C","'",")",",","(","'","Y","'",",","'D","'",")","]","For","Series",",","it","also","holds","scol","to","represent","the","column",".",">",">",">","psseries","=","psdf1.B",">",">",">","psseries","A","1","5","2","6","3","7","4","8","Name",":","B",",","dtype",":","int64",">",">",">","internal","=","psseries._internal",">",">",">","internal.spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+ELLIPSIS","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+","|__index_level_0__|","A|","B|","C|","D|","E|__natural_order__|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+","|","0|","1|","5|","9|","13|","17|","...","|","|","1|","2|","6|","10|","14|","18|","...","|","|","2|","3|","7|","11|","15|","19|","...","|","|","3|","4|","8|","12|","16|","20|","...","|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+",">",">",">","internal.data_spark_column_names","[","'","B","'","]",">",">",">","internal.index_spark_column_names","[","'","A","'","]",">",">",">","internal.spark_column_names","[","'","A","'",",","'","B","'","]",">",">",">","internal.index_names","[","(","'","A","'",",",")","]",">",">",">","internal.data_fields","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","B","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.index_fields","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","A","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.to_internal_spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+","--","-+","--","-+","|","A|","B|","+","--","-+","--","-+","|","1|","5|","|","2|","6|","|","3|","7|","|","4|","8|","+","--","-+","--","-+",">",">",">","internal.to_pandas_frame","#","doctest",":","+NORMALIZE_WHITESPACE","B","A","1","5","2","6","3","7","4","8"]
docstr: Copy the immutable InternalFrame.
  
  :param spark_frame: the new Spark DataFrame. If not specified, the original one is used.
  :param index_spark_columns: the list of Spark Column.
                              If not specified, the original ones are used.
  :param index_names: the index names. If not specified, the original ones are used.
  :param index_fields: the new InternalFields for the index columns.
                       If not specified, the original metadata are used.
  :param column_labels: the new column labels. If not specified, the original ones are used.
  :param data_spark_columns: the new Spark Columns.
                             If not specified, the original ones are used.
  :param data_fields: the new InternalFields for the data columns.
                      If not specified, the original metadata are used.
  :param column_label_names: the new names of the column index levels.
                             If not specified, the original ones are used.
  :return: the copied immutable InternalFrame.
docstr_tok: ["Copy","the","immutable","InternalFrame",".",":","param","spark_frame",":","the","new","Spark","DataFrame",".","If","not","specified",",","the","original","one","is","used",".",":","param","index_spark_columns",":","the","list","of","Spark","Column",".","If","not","specified",",","the","original","ones","are","used",".",":","param","index_names",":","the","index","names",".","If","not","specified",",","the","original","ones","are","used",".",":","param","index_fields",":","the","new","InternalFields","for","the","index","columns",".","If","not","specified",",","the","original","metadata","are","used",".",":","param","column_labels",":","the","new","column","labels",".","If","not","specified",",","the","original","ones","are","used",".",":","param","data_spark_columns",":","the","new","Spark","Columns",".","If","not","specified",",","the","original","ones","are","used",".",":","param","data_fields",":","the","new","InternalFields","for","the","data","columns",".","If","not","specified",",","the","original","metadata","are","used",".",":","param","column_label_names",":","the","new","names","of","the","column","index","levels",".","If","not","specified",",","the","original","ones","are","used",".",":","return",":","the","copied","immutable","InternalFrame","."]
code: def copy(
      self,
      *,
      spark_frame: Union[PySparkDataFrame, _NoValueType] = _NoValue,
      index_spark_columns: Union[
          List[PySparkColumn], _NoValueType
      ] = _NoValue,
      index_names: Union[
          Optional[List[Optional[Label]]], _NoValueType
      ] = _NoValue,
      index_fields: Union[
          Optional[List[InternalField]], _NoValueType
      ] = _NoValue,
      column_labels: Union[
          Optional[List[Label]], _NoValueType
      ] = _NoValue,
      data_spark_columns: Union[
          Optional[List[PySparkColumn]], _NoValueType
      ] = _NoValue,
      data_fields: Union[
          Optional[List[InternalField]], _NoValueType
      ] = _NoValue,
      column_label_names: Union[
          Optional[List[Optional[Label]]], _NoValueType
      ] = _NoValue,
  ) -> "InternalFrame":
      """
      Copy the immutable InternalFrame.
  
      :param spark_frame: the new Spark DataFrame. If not specified, the original one is used.
      :param index_spark_columns: the list of Spark Column.
                                  If not specified, the original ones are used.
      :param index_names: the index names. If not specified, the original ones are used.
      :param index_fields: the new InternalFields for the index columns.
                           If not specified, the original metadata are used.
      :param column_labels: the new column labels. If not specified, the original ones are used.
      :param data_spark_columns: the new Spark Columns.
                                 If not specified, the original ones are used.
      :param data_fields: the new InternalFields for the data columns.
                          If not specified, the original metadata are used.
      :param column_label_names: the new names of the column index levels.
                                 If not specified, the original ones are used.
      :return: the copied immutable InternalFrame.
      """
      if spark_frame is _NoValue:
          spark_frame = self.spark_frame
      if index_spark_columns is _NoValue:
          index_spark_columns = self.index_spark_columns
      if index_names is _NoValue:
          index_names = self.index_names
      if index_fields is _NoValue:
          index_fields = self.index_fields
      if column_labels is _NoValue:
          column_labels = self.column_labels
      if data_spark_columns is _NoValue:
          data_spark_columns = self.data_spark_columns
      if data_fields is _NoValue:
          data_fields = self.data_fields
      if column_label_names is _NoValue:
          column_label_names = self.column_label_names
      return InternalFrame(
          spark_frame=cast(PySparkDataFrame, spark_frame),
          index_spark_columns=cast(
              List[PySparkColumn], index_spark_columns
          ),
          index_names=cast(
              Optional[List[Optional[Label]]], index_names
          ),
          index_fields=cast(
              Optional[List[InternalField]], index_fields
          ),
          column_labels=cast(Optional[List[Label]], column_labels),
          data_spark_columns=cast(
              Optional[List[PySparkColumn]], data_spark_columns
          ),
          data_fields=cast(Optional[List[InternalField]], data_fields),
          column_label_names=cast(
              Optional[List[Optional[Label]]], column_label_names
          ),
      )
  
imports: import re
  from typing import Any, Dict, List, Optional, Sequence, Tuple, Union, TYPE_CHECKING, cast
  import numpy
  import pandas
  from pandas.api.types import CategoricalDtype
  from pyspark._globals import _NoValue, _NoValueType
  from pyspark.sql import functions, Column, DataFrame, Window
  from pyspark.sql.types import BooleanType, DataType, LongType, StructField, StructType, StringType
  from pyspark.sql.utils import is_timestamp_ntz_preferred
  from pyspark.sql.utils import is_remote, get_column_class, get_dataframe_class
  from pyspark import pandas
  from pyspark.pandas._typing import Label
  from pyspark.pandas.spark.utils import as_nullable_spark_type, force_decimal_precision_scale
  from pyspark.pandas.data_type_ops.base import DataTypeOps
  from pyspark.pandas.typedef import Dtype, as_spark_type, extension_dtypes, infer_pd_series_spark_type, spark_type_to_pandas_dtype
  from pyspark.pandas.utils import column_labels_level, default_session, is_name_like_tuple, is_testing, lazy_property, name_like_string, scol_for, spark_column_equals
label_desc: Use image node_id 35 for calling the InternalFrame obj's underlying member method code with example usage: obj.copy() and returns: InternalFrame