class_docstr: The internal immutable DataFrame which manages Spark DataFrame and column names and index
  information.
  
  .. note:: this is an internal class. It is not supposed to be exposed to users and users
      should not directly access to it.
  
  The internal immutable DataFrame represents the index information for a DataFrame it belongs to.
  For instance, if we have a pandas-on-Spark DataFrame as below, pandas DataFrame does not
  store the index as columns.
  
  >>> psdf = ps.DataFrame({
  ...     'A': [1, 2, 3, 4],
  ...     'B': [5, 6, 7, 8],
  ...     'C': [9, 10, 11, 12],
  ...     'D': [13, 14, 15, 16],
  ...     'E': [17, 18, 19, 20]}, columns = ['A', 'B', 'C', 'D', 'E'])
  >>> psdf  # doctest: +NORMALIZE_WHITESPACE
     A  B   C   D   E
  0  1  5   9  13  17
  1  2  6  10  14  18
  2  3  7  11  15  19
  3  4  8  12  16  20
  
  However, all columns including index column are also stored in Spark DataFrame internally
  as below.
  
  >>> psdf._internal.to_internal_spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE
  +-----------------+---+---+---+---+---+
  |__index_level_0__|  A|  B|  C|  D|  E|
  +-----------------+---+---+---+---+---+
  |                0|  1|  5|  9| 13| 17|
  |                1|  2|  6| 10| 14| 18|
  |                2|  3|  7| 11| 15| 19|
  |                3|  4|  8| 12| 16| 20|
  +-----------------+---+---+---+---+---+
  
  To fill this gap, the current metadata is used by mapping Spark's internal column
  to pandas-on-Spark's index. See the method below:
  
  * `spark_frame` represents the internal Spark DataFrame
  
  * `data_spark_column_names` represents non-indexing Spark column names
  
  * `data_spark_columns` represents non-indexing Spark columns
  
  * `data_fields` represents non-indexing InternalFields
  
  * `index_spark_column_names` represents internal index Spark column names
  
  * `index_spark_columns` represents internal index Spark columns
  
  * `index_fields` represents index InternalFields
  
  * `spark_column_names` represents all columns
  
  * `index_names` represents the external index name as a label
  
  * `to_internal_spark_frame` represents Spark DataFrame derived by the metadata. Includes index.
  
  * `to_pandas_frame` represents pandas DataFrame derived by the metadata
  
  >>> internal = psdf._internal
  >>> internal.spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
  +-----------------+---+---+---+---+---+-----------------+
  |__index_level_0__|  A|  B|  C|  D|  E|__natural_order__|
  +-----------------+---+---+---+---+---+-----------------+
  |                0|  1|  5|  9| 13| 17|              ...|
  |                1|  2|  6| 10| 14| 18|              ...|
  |                2|  3|  7| 11| 15| 19|              ...|
  |                3|  4|  8| 12| 16| 20|              ...|
  +-----------------+---+---+---+---+---+-----------------+
  >>> internal.data_spark_column_names
  ['A', 'B', 'C', 'D', 'E']
  >>> internal.index_spark_column_names
  ['__index_level_0__']
  >>> internal.spark_column_names
  ['__index_level_0__', 'A', 'B', 'C', 'D', 'E']
  >>> internal.index_names
  [None]
  >>> internal.data_fields    # doctest: +NORMALIZE_WHITESPACE
  [InternalField(dtype=int64, struct_field=StructField('A', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('B', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('C', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('D', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('E', LongType(), False))]
  >>> internal.index_fields
  [InternalField(dtype=int64, struct_field=StructField('__index_level_0__', LongType(), False))]
  >>> internal.to_internal_spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE
  +-----------------+---+---+---+---+---+
  |__index_level_0__|  A|  B|  C|  D|  E|
  +-----------------+---+---+---+---+---+
  |                0|  1|  5|  9| 13| 17|
  |                1|  2|  6| 10| 14| 18|
  |                2|  3|  7| 11| 15| 19|
  |                3|  4|  8| 12| 16| 20|
  +-----------------+---+---+---+---+---+
  >>> internal.to_pandas_frame
     A  B   C   D   E
  0  1  5   9  13  17
  1  2  6  10  14  18
  2  3  7  11  15  19
  3  4  8  12  16  20
  
  In case that index is set to one of the existing columns as below:
  
  >>> psdf1 = psdf.set_index("A")
  >>> psdf1  # doctest: +NORMALIZE_WHITESPACE
     B   C   D   E
  A
  1  5   9  13  17
  2  6  10  14  18
  3  7  11  15  19
  4  8  12  16  20
  
  >>> psdf1._internal.to_internal_spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE
  +---+---+---+---+---+
  |  A|  B|  C|  D|  E|
  +---+---+---+---+---+
  |  1|  5|  9| 13| 17|
  |  2|  6| 10| 14| 18|
  |  3|  7| 11| 15| 19|
  |  4|  8| 12| 16| 20|
  +---+---+---+---+---+
  
  >>> internal = psdf1._internal
  >>> internal.spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
  +-----------------+---+---+---+---+---+-----------------+
  |__index_level_0__|  A|  B|  C|  D|  E|__natural_order__|
  +-----------------+---+---+---+---+---+-----------------+
  |                0|  1|  5|  9| 13| 17|              ...|
  |                1|  2|  6| 10| 14| 18|              ...|
  |                2|  3|  7| 11| 15| 19|              ...|
  |                3|  4|  8| 12| 16| 20|              ...|
  +-----------------+---+---+---+---+---+-----------------+
  >>> internal.data_spark_column_names
  ['B', 'C', 'D', 'E']
  >>> internal.index_spark_column_names
  ['A']
  >>> internal.spark_column_names
  ['A', 'B', 'C', 'D', 'E']
  >>> internal.index_names
  [('A',)]
  >>> internal.data_fields  # doctest: +NORMALIZE_WHITESPACE
  [InternalField(dtype=int64, struct_field=StructField('B', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('C', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('D', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('E', LongType(), False))]
  >>> internal.index_fields
  [InternalField(dtype=int64, struct_field=StructField('A', LongType(), False))]
  >>> internal.to_internal_spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE
  +---+---+---+---+---+
  |  A|  B|  C|  D|  E|
  +---+---+---+---+---+
  |  1|  5|  9| 13| 17|
  |  2|  6| 10| 14| 18|
  |  3|  7| 11| 15| 19|
  |  4|  8| 12| 16| 20|
  +---+---+---+---+---+
  >>> internal.to_pandas_frame  # doctest: +NORMALIZE_WHITESPACE
     B   C   D   E
  A
  1  5   9  13  17
  2  6  10  14  18
  3  7  11  15  19
  4  8  12  16  20
  
  In case that index becomes a multi index as below:
  
  >>> psdf2 = psdf.set_index("A", append=True)
  >>> psdf2  # doctest: +NORMALIZE_WHITESPACE
       B   C   D   E
    A
  0 1  5   9  13  17
  1 2  6  10  14  18
  2 3  7  11  15  19
  3 4  8  12  16  20
  
  >>> psdf2._internal.to_internal_spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE
  +-----------------+---+---+---+---+---+
  |__index_level_0__|  A|  B|  C|  D|  E|
  +-----------------+---+---+---+---+---+
  |                0|  1|  5|  9| 13| 17|
  |                1|  2|  6| 10| 14| 18|
  |                2|  3|  7| 11| 15| 19|
  |                3|  4|  8| 12| 16| 20|
  +-----------------+---+---+---+---+---+
  
  >>> internal = psdf2._internal
  >>> internal.spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
  +-----------------+---+---+---+---+---+-----------------+
  |__index_level_0__|  A|  B|  C|  D|  E|__natural_order__|
  +-----------------+---+---+---+---+---+-----------------+
  |                0|  1|  5|  9| 13| 17|              ...|
  |                1|  2|  6| 10| 14| 18|              ...|
  |                2|  3|  7| 11| 15| 19|              ...|
  |                3|  4|  8| 12| 16| 20|              ...|
  +-----------------+---+---+---+---+---+-----------------+
  >>> internal.data_spark_column_names
  ['B', 'C', 'D', 'E']
  >>> internal.index_spark_column_names
  ['__index_level_0__', 'A']
  >>> internal.spark_column_names
  ['__index_level_0__', 'A', 'B', 'C', 'D', 'E']
  >>> internal.index_names
  [None, ('A',)]
  >>> internal.data_fields  # doctest: +NORMALIZE_WHITESPACE
  [InternalField(dtype=int64, struct_field=StructField('B', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('C', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('D', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('E', LongType(), False))]
  >>> internal.index_fields  # doctest: +NORMALIZE_WHITESPACE
  [InternalField(dtype=int64, struct_field=StructField('__index_level_0__', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('A', LongType(), False))]
  >>> internal.to_internal_spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE
  +-----------------+---+---+---+---+---+
  |__index_level_0__|  A|  B|  C|  D|  E|
  +-----------------+---+---+---+---+---+
  |                0|  1|  5|  9| 13| 17|
  |                1|  2|  6| 10| 14| 18|
  |                2|  3|  7| 11| 15| 19|
  |                3|  4|  8| 12| 16| 20|
  +-----------------+---+---+---+---+---+
  >>> internal.to_pandas_frame  # doctest: +NORMALIZE_WHITESPACE
       B   C   D   E
    A
  0 1  5   9  13  17
  1 2  6  10  14  18
  2 3  7  11  15  19
  3 4  8  12  16  20
  
  For multi-level columns, it also holds column_labels
  
  >>> columns = pd.MultiIndex.from_tuples([('X', 'A'), ('X', 'B'),
  ...                                      ('Y', 'C'), ('Y', 'D')])
  >>> psdf3 = ps.DataFrame([
  ...     [1, 2, 3, 4],
  ...     [5, 6, 7, 8],
  ...     [9, 10, 11, 12],
  ...     [13, 14, 15, 16],
  ...     [17, 18, 19, 20]], columns = columns)
  >>> psdf3  # doctest: +NORMALIZE_WHITESPACE
      X       Y
      A   B   C   D
  0   1   2   3   4
  1   5   6   7   8
  2   9  10  11  12
  3  13  14  15  16
  4  17  18  19  20
  
  >>> internal = psdf3._internal
  >>> internal.spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
  +-----------------+------+------+------+------+-----------------+
  |__index_level_0__|(X, A)|(X, B)|(Y, C)|(Y, D)|__natural_order__|
  +-----------------+------+------+------+------+-----------------+
  |                0|     1|     2|     3|     4|              ...|
  |                1|     5|     6|     7|     8|              ...|
  |                2|     9|    10|    11|    12|              ...|
  |                3|    13|    14|    15|    16|              ...|
  |                4|    17|    18|    19|    20|              ...|
  +-----------------+------+------+------+------+-----------------+
  >>> internal.data_spark_column_names
  ['(X, A)', '(X, B)', '(Y, C)', '(Y, D)']
  >>> internal.column_labels
  [('X', 'A'), ('X', 'B'), ('Y', 'C'), ('Y', 'D')]
  
  For Series, it also holds scol to represent the column.
  
  >>> psseries = psdf1.B
  >>> psseries
  A
  1    5
  2    6
  3    7
  4    8
  Name: B, dtype: int64
  
  >>> internal = psseries._internal
  >>> internal.spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
  +-----------------+---+---+---+---+---+-----------------+
  |__index_level_0__|  A|  B|  C|  D|  E|__natural_order__|
  +-----------------+---+---+---+---+---+-----------------+
  |                0|  1|  5|  9| 13| 17|              ...|
  |                1|  2|  6| 10| 14| 18|              ...|
  |                2|  3|  7| 11| 15| 19|              ...|
  |                3|  4|  8| 12| 16| 20|              ...|
  +-----------------+---+---+---+---+---+-----------------+
  >>> internal.data_spark_column_names
  ['B']
  >>> internal.index_spark_column_names
  ['A']
  >>> internal.spark_column_names
  ['A', 'B']
  >>> internal.index_names
  [('A',)]
  >>> internal.data_fields
  [InternalField(dtype=int64, struct_field=StructField('B', LongType(), False))]
  >>> internal.index_fields
  [InternalField(dtype=int64, struct_field=StructField('A', LongType(), False))]
  >>> internal.to_internal_spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE
  +---+---+
  |  A|  B|
  +---+---+
  |  1|  5|
  |  2|  6|
  |  3|  7|
  |  4|  8|
  +---+---+
  >>> internal.to_pandas_frame  # doctest: +NORMALIZE_WHITESPACE
     B
  A
  1  5
  2  6
  3  7
  4  8
class_docstr_tok: ["The","internal","immutable","DataFrame","which","manages","Spark","DataFrame","and","column","names","and","index","information",".","..","note",":",":","this","is","an","internal","class",".","It","is","not","supposed","to","be","exposed","to","users","and","users","should","not","directly","access","to","it",".","The","internal","immutable","DataFrame","represents","the","index","information","for","a","DataFrame","it","belongs","to",".","For","instance",",","if","we","have","a","pandas-on-Spark","DataFrame","as","below",",","pandas","DataFrame","does","not","store","the","index","as","columns",".",">",">",">","psdf","=","ps.DataFrame","(","{","...","'","A","'",":","[","1",",","2",",","3",",","4","]",",","...","'","B","'",":","[","5",",","6",",","7",",","8","]",",","...","'","C","'",":","[","9",",","10",",","11",",","12","]",",","...","'D","'",":","[","13",",","14",",","15",",","16","]",",","...","'","E","'",":","[","17",",","18",",","19",",","20","]","}",",","columns","=","[","'","A","'",",","'","B","'",",","'","C","'",",","'D","'",",","'","E","'","]",")",">",">",">","psdf","#","doctest",":","+NORMALIZE_WHITESPACE","A","B","C","D","E","0","1","5","9","13","17","1","2","6","10","14","18","2","3","7","11","15","19","3","4","8","12","16","20","However",",","all","columns","including","index","column","are","also","stored","in","Spark","DataFrame","internally","as","below",".",">",">",">","psdf._internal.to_internal_spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","|__index_level_0__|","A|","B|","C|","D|","E|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","|","0|","1|","5|","9|","13|","17|","|","1|","2|","6|","10|","14|","18|","|","2|","3|","7|","11|","15|","19|","|","3|","4|","8|","12|","16|","20|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","To","fill","this","gap",",","the","current","metadata","is","used","by","mapping","Spark","'s","internal","column","to","pandas-on-Spark","'s","index",".","See","the","method","below",":","*","`","spark_frame","`","represents","the","internal","Spark","DataFrame","*","`","data_spark_column_names","`","represents","non-indexing","Spark","column","names","*","`","data_spark_columns","`","represents","non-indexing","Spark","columns","*","`","data_fields","`","represents","non-indexing","InternalFields","*","`","index_spark_column_names","`","represents","internal","index","Spark","column","names","*","`","index_spark_columns","`","represents","internal","index","Spark","columns","*","`","index_fields","`","represents","index","InternalFields","*","`","spark_column_names","`","represents","all","columns","*","`","index_names","`","represents","the","external","index","name","as","a","label","*","`","to_internal_spark_frame","`","represents","Spark","DataFrame","derived","by","the","metadata",".","Includes","index",".","*","`","to_pandas_frame","`","represents","pandas","DataFrame","derived","by","the","metadata",">",">",">","internal","=","psdf._internal",">",">",">","internal.spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+ELLIPSIS","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+","|__index_level_0__|","A|","B|","C|","D|","E|__natural_order__|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+","|","0|","1|","5|","9|","13|","17|","...","|","|","1|","2|","6|","10|","14|","18|","...","|","|","2|","3|","7|","11|","15|","19|","...","|","|","3|","4|","8|","12|","16|","20|","...","|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+",">",">",">","internal.data_spark_column_names","[","'","A","'",",","'","B","'",",","'","C","'",",","'D","'",",","'","E","'","]",">",">",">","internal.index_spark_column_names","[","'__index_level_0__","'","]",">",">",">","internal.spark_column_names","[","'__index_level_0__","'",",","'","A","'",",","'","B","'",",","'","C","'",",","'D","'",",","'","E","'","]",">",">",">","internal.index_names","[","None","]",">",">",">","internal.data_fields","#","doctest",":","+NORMALIZE_WHITESPACE","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","A","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","B","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","C","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'D","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","E","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.index_fields","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'__index_level_0__","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.to_internal_spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","|__index_level_0__|","A|","B|","C|","D|","E|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","|","0|","1|","5|","9|","13|","17|","|","1|","2|","6|","10|","14|","18|","|","2|","3|","7|","11|","15|","19|","|","3|","4|","8|","12|","16|","20|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+",">",">",">","internal.to_pandas_frame","A","B","C","D","E","0","1","5","9","13","17","1","2","6","10","14","18","2","3","7","11","15","19","3","4","8","12","16","20","In","case","that","index","is","set","to","one","of","the","existing","columns","as","below",":",">",">",">","psdf1","=","psdf.set_index","(","``","A","''",")",">",">",">","psdf1","#","doctest",":","+NORMALIZE_WHITESPACE","B","C","D","E","A","1","5","9","13","17","2","6","10","14","18","3","7","11","15","19","4","8","12","16","20",">",">",">","psdf1._internal.to_internal_spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+","--","-+","--","-+","--","-+","--","-+","--","-+","|","A|","B|","C|","D|","E|","+","--","-+","--","-+","--","-+","--","-+","--","-+","|","1|","5|","9|","13|","17|","|","2|","6|","10|","14|","18|","|","3|","7|","11|","15|","19|","|","4|","8|","12|","16|","20|","+","--","-+","--","-+","--","-+","--","-+","--","-+",">",">",">","internal","=","psdf1._internal",">",">",">","internal.spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+ELLIPSIS","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+","|__index_level_0__|","A|","B|","C|","D|","E|__natural_order__|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+","|","0|","1|","5|","9|","13|","17|","...","|","|","1|","2|","6|","10|","14|","18|","...","|","|","2|","3|","7|","11|","15|","19|","...","|","|","3|","4|","8|","12|","16|","20|","...","|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+",">",">",">","internal.data_spark_column_names","[","'","B","'",",","'","C","'",",","'D","'",",","'","E","'","]",">",">",">","internal.index_spark_column_names","[","'","A","'","]",">",">",">","internal.spark_column_names","[","'","A","'",",","'","B","'",",","'","C","'",",","'D","'",",","'","E","'","]",">",">",">","internal.index_names","[","(","'","A","'",",",")","]",">",">",">","internal.data_fields","#","doctest",":","+NORMALIZE_WHITESPACE","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","B","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","C","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'D","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","E","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.index_fields","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","A","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.to_internal_spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+","--","-+","--","-+","--","-+","--","-+","--","-+","|","A|","B|","C|","D|","E|","+","--","-+","--","-+","--","-+","--","-+","--","-+","|","1|","5|","9|","13|","17|","|","2|","6|","10|","14|","18|","|","3|","7|","11|","15|","19|","|","4|","8|","12|","16|","20|","+","--","-+","--","-+","--","-+","--","-+","--","-+",">",">",">","internal.to_pandas_frame","#","doctest",":","+NORMALIZE_WHITESPACE","B","C","D","E","A","1","5","9","13","17","2","6","10","14","18","3","7","11","15","19","4","8","12","16","20","In","case","that","index","becomes","a","multi","index","as","below",":",">",">",">","psdf2","=","psdf.set_index","(","``","A","''",",","append=True",")",">",">",">","psdf2","#","doctest",":","+NORMALIZE_WHITESPACE","B","C","D","E","A","0","1","5","9","13","17","1","2","6","10","14","18","2","3","7","11","15","19","3","4","8","12","16","20",">",">",">","psdf2._internal.to_internal_spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","|__index_level_0__|","A|","B|","C|","D|","E|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","|","0|","1|","5|","9|","13|","17|","|","1|","2|","6|","10|","14|","18|","|","2|","3|","7|","11|","15|","19|","|","3|","4|","8|","12|","16|","20|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+",">",">",">","internal","=","psdf2._internal",">",">",">","internal.spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+ELLIPSIS","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+","|__index_level_0__|","A|","B|","C|","D|","E|__natural_order__|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+","|","0|","1|","5|","9|","13|","17|","...","|","|","1|","2|","6|","10|","14|","18|","...","|","|","2|","3|","7|","11|","15|","19|","...","|","|","3|","4|","8|","12|","16|","20|","...","|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+",">",">",">","internal.data_spark_column_names","[","'","B","'",",","'","C","'",",","'D","'",",","'","E","'","]",">",">",">","internal.index_spark_column_names","[","'__index_level_0__","'",",","'","A","'","]",">",">",">","internal.spark_column_names","[","'__index_level_0__","'",",","'","A","'",",","'","B","'",",","'","C","'",",","'D","'",",","'","E","'","]",">",">",">","internal.index_names","[","None",",","(","'","A","'",",",")","]",">",">",">","internal.data_fields","#","doctest",":","+NORMALIZE_WHITESPACE","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","B","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","C","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'D","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","E","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.index_fields","#","doctest",":","+NORMALIZE_WHITESPACE","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'__index_level_0__","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","A","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.to_internal_spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","|__index_level_0__|","A|","B|","C|","D|","E|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","|","0|","1|","5|","9|","13|","17|","|","1|","2|","6|","10|","14|","18|","|","2|","3|","7|","11|","15|","19|","|","3|","4|","8|","12|","16|","20|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+",">",">",">","internal.to_pandas_frame","#","doctest",":","+NORMALIZE_WHITESPACE","B","C","D","E","A","0","1","5","9","13","17","1","2","6","10","14","18","2","3","7","11","15","19","3","4","8","12","16","20","For","multi-level","columns",",","it","also","holds","column_labels",">",">",">","columns","=","pd.MultiIndex.from_tuples","(","[","(","'","X","'",",","'","A","'",")",",","(","'","X","'",",","'","B","'",")",",","...","(","'","Y","'",",","'","C","'",")",",","(","'","Y","'",",","'D","'",")","]",")",">",">",">","psdf3","=","ps.DataFrame","(","[","...","[","1",",","2",",","3",",","4","]",",","...","[","5",",","6",",","7",",","8","]",",","...","[","9",",","10",",","11",",","12","]",",","...","[","13",",","14",",","15",",","16","]",",","...","[","17",",","18",",","19",",","20","]","]",",","columns","=","columns",")",">",">",">","psdf3","#","doctest",":","+NORMALIZE_WHITESPACE","X","Y","A","B","C","D","0","1","2","3","4","1","5","6","7","8","2","9","10","11","12","3","13","14","15","16","4","17","18","19","20",">",">",">","internal","=","psdf3._internal",">",">",">","internal.spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+ELLIPSIS","+","--","--","--","--","--","--","--","--","-+","--","--","--","+","--","--","--","+","--","--","--","+","--","--","--","+","--","--","--","--","--","--","--","--","-+","|__index_level_0__|","(","X",",","A",")","|","(","X",",","B",")","|","(","Y",",","C",")","|","(","Y",",","D",")","|__natural_order__|","+","--","--","--","--","--","--","--","--","-+","--","--","--","+","--","--","--","+","--","--","--","+","--","--","--","+","--","--","--","--","--","--","--","--","-+","|","0|","1|","2|","3|","4|","...","|","|","1|","5|","6|","7|","8|","...","|","|","2|","9|","10|","11|","12|","...","|","|","3|","13|","14|","15|","16|","...","|","|","4|","17|","18|","19|","20|","...","|","+","--","--","--","--","--","--","--","--","-+","--","--","--","+","--","--","--","+","--","--","--","+","--","--","--","+","--","--","--","--","--","--","--","--","-+",">",">",">","internal.data_spark_column_names","[","'","(","X",",","A",")","'",",","'","(","X",",","B",")","'",",","'","(","Y",",","C",")","'",",","'","(","Y",",","D",")","'","]",">",">",">","internal.column_labels","[","(","'","X","'",",","'","A","'",")",",","(","'","X","'",",","'","B","'",")",",","(","'","Y","'",",","'","C","'",")",",","(","'","Y","'",",","'D","'",")","]","For","Series",",","it","also","holds","scol","to","represent","the","column",".",">",">",">","psseries","=","psdf1.B",">",">",">","psseries","A","1","5","2","6","3","7","4","8","Name",":","B",",","dtype",":","int64",">",">",">","internal","=","psseries._internal",">",">",">","internal.spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+ELLIPSIS","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+","|__index_level_0__|","A|","B|","C|","D|","E|__natural_order__|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+","|","0|","1|","5|","9|","13|","17|","...","|","|","1|","2|","6|","10|","14|","18|","...","|","|","2|","3|","7|","11|","15|","19|","...","|","|","3|","4|","8|","12|","16|","20|","...","|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+",">",">",">","internal.data_spark_column_names","[","'","B","'","]",">",">",">","internal.index_spark_column_names","[","'","A","'","]",">",">",">","internal.spark_column_names","[","'","A","'",",","'","B","'","]",">",">",">","internal.index_names","[","(","'","A","'",",",")","]",">",">",">","internal.data_fields","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","B","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.index_fields","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","A","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.to_internal_spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+","--","-+","--","-+","|","A|","B|","+","--","-+","--","-+","|","1|","5|","|","2|","6|","|","3|","7|","|","4|","8|","+","--","-+","--","-+",">",">",">","internal.to_pandas_frame","#","doctest",":","+NORMALIZE_WHITESPACE","B","A","1","5","2","6","3","7","4","8"]
docstr: Create a new internal immutable DataFrame to manage Spark DataFrame, column fields and
  index fields and names.
  
  :param spark_frame: Spark DataFrame to be managed.
  :param index_spark_columns: list of Spark Column
                              Spark Columns for the index.
  :param index_names: list of tuples
                      the index names.
  :param index_fields: list of InternalField
                       the InternalFields for the index columns
  :param column_labels: list of tuples with the same length
                        The multi-level values in the tuples.
  :param data_spark_columns: list of Spark Column
                             Spark Columns to appear as columns. If this is None, calculated
                             from spark_frame.
  :param data_fields: list of InternalField
                      the InternalFields for the data columns
  :param column_label_names: Names for each of the column index levels.
  
  See the examples below to refer what each parameter means.
  
  >>> column_labels = pd.MultiIndex.from_tuples(
  ...     [('a', 'x'), ('a', 'y'), ('b', 'z')], names=["column_labels_a", "column_labels_b"])
  >>> row_index = pd.MultiIndex.from_tuples(
  ...     [('foo', 'bar'), ('foo', 'bar'), ('zoo', 'bar')],
  ...     names=["row_index_a", "row_index_b"])
  >>> psdf = ps.DataFrame(
  ...     [[1, 2, 3], [4, 5, 6], [7, 8, 9]], index=row_index, columns=column_labels)
  >>> psdf.set_index(('a', 'x'), append=True, inplace=True)
  >>> psdf  # doctest: +NORMALIZE_WHITESPACE
  column_labels_a                  a  b
  column_labels_b                  y  z
  row_index_a row_index_b (a, x)
  foo         bar         1       2  3
                          4       5  6
  zoo         bar         7       8  9
  
  >>> internal = psdf._internal
  
  >>> internal.spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
  +-----------------+-----------------+------+------+------+...
  |__index_level_0__|__index_level_1__|(a, x)|(a, y)|(b, z)|...
  +-----------------+-----------------+------+------+------+...
  |              foo|              bar|     1|     2|     3|...
  |              foo|              bar|     4|     5|     6|...
  |              zoo|              bar|     7|     8|     9|...
  +-----------------+-----------------+------+------+------+...
  
  >>> internal.index_spark_columns  # doctest: +SKIP
  [Column<'__index_level_0__'>, Column<'__index_level_1__'>, Column<'(a, x)'>]
  
  >>> internal.index_names
  [('row_index_a',), ('row_index_b',), ('a', 'x')]
  
  >>> internal.index_fields  # doctest: +NORMALIZE_WHITESPACE
  [InternalField(dtype=object,
      struct_field=StructField('__index_level_0__', StringType(), False)),
   InternalField(dtype=object,
      struct_field=StructField('__index_level_1__', StringType(), False)),
   InternalField(dtype=int64,
      struct_field=StructField('(a, x)', LongType(), False))]
  
  >>> internal.column_labels
  [('a', 'y'), ('b', 'z')]
  
  >>> internal.data_spark_columns  # doctest: +SKIP
  [Column<'(a, y)'>, Column<'(b, z)'>]
  
  >>> internal.data_fields  # doctest: +NORMALIZE_WHITESPACE
  [InternalField(dtype=int64, struct_field=StructField('(a, y)', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('(b, z)', LongType(), False))]
  
  >>> internal.column_label_names
  [('column_labels_a',), ('column_labels_b',)]
docstr_tok: ["Create","a","new","internal","immutable","DataFrame","to","manage","Spark","DataFrame",",","column","fields","and","index","fields","and","names",".",":","param","spark_frame",":","Spark","DataFrame","to","be","managed",".",":","param","index_spark_columns",":","list","of","Spark","Column","Spark","Columns","for","the","index",".",":","param","index_names",":","list","of","tuples","the","index","names",".",":","param","index_fields",":","list","of","InternalField","the","InternalFields","for","the","index","columns",":","param","column_labels",":","list","of","tuples","with","the","same","length","The","multi-level","values","in","the","tuples",".",":","param","data_spark_columns",":","list","of","Spark","Column","Spark","Columns","to","appear","as","columns",".","If","this","is","None",",","calculated","from","spark_frame",".",":","param","data_fields",":","list","of","InternalField","the","InternalFields","for","the","data","columns",":","param","column_label_names",":","Names","for","each","of","the","column","index","levels",".","See","the","examples","below","to","refer","what","each","parameter","means",".",">",">",">","column_labels","=","pd.MultiIndex.from_tuples","(","...","[","(","'","a","'",",","'","x","'",")",",","(","'","a","'",",","'","y","'",")",",","(","'","b","'",",","'","z","'",")","]",",","names=","[","``","column_labels_a","''",",","``","column_labels_b","''","]",")",">",">",">","row_index","=","pd.MultiIndex.from_tuples","(","...","[","(","'foo","'",",","'bar","'",")",",","(","'foo","'",",","'bar","'",")",",","(","'zoo","'",",","'bar","'",")","]",",","...","names=","[","``","row_index_a","''",",","``","row_index_b","''","]",")",">",">",">","psdf","=","ps.DataFrame","(","...","[","[","1",",","2",",","3","]",",","[","4",",","5",",","6","]",",","[","7",",","8",",","9","]","]",",","index=row_index",",","columns=column_labels",")",">",">",">","psdf.set_index","(","(","'","a","'",",","'","x","'",")",",","append=True",",","inplace=True",")",">",">",">","psdf","#","doctest",":","+NORMALIZE_WHITESPACE","column_labels_a","a","b","column_labels_b","y","z","row_index_a","row_index_b","(","a",",","x",")","foo","bar","1","2","3","4","5","6","zoo","bar","7","8","9",">",">",">","internal","=","psdf._internal",">",">",">","internal.spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+ELLIPSIS","+","--","--","--","--","--","--","--","--","-+","--","--","--","--","--","--","--","--","-+","--","--","--","+","--","--","--","+","--","--","--","+","...","|__index_level_0__|__index_level_1__|","(","a",",","x",")","|","(","a",",","y",")","|","(","b",",","z",")","|","...","+","--","--","--","--","--","--","--","--","-+","--","--","--","--","--","--","--","--","-+","--","--","--","+","--","--","--","+","--","--","--","+","...","|","foo|","bar|","1|","2|","3|","...","|","foo|","bar|","4|","5|","6|","...","|","zoo|","bar|","7|","8|","9|","...","+","--","--","--","--","--","--","--","--","-+","--","--","--","--","--","--","--","--","-+","--","--","--","+","--","--","--","+","--","--","--","+","...",">",">",">","internal.index_spark_columns","#","doctest",":","+SKIP","[","Column","<","'__index_level_0__","'",">",",","Column","<","'__index_level_1__","'",">",",","Column","<","'","(","a",",","x",")","'",">","]",">",">",">","internal.index_names","[","(","'row_index_a","'",",",")",",","(","'row_index_b","'",",",")",",","(","'","a","'",",","'","x","'",")","]",">",">",">","internal.index_fields","#","doctest",":","+NORMALIZE_WHITESPACE","[","InternalField","(","dtype=object",",","struct_field=StructField","(","'__index_level_0__","'",",","StringType","(",")",",","False",")",")",",","InternalField","(","dtype=object",",","struct_field=StructField","(","'__index_level_1__","'",",","StringType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","(","a",",","x",")","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.column_labels","[","(","'","a","'",",","'","y","'",")",",","(","'","b","'",",","'","z","'",")","]",">",">",">","internal.data_spark_columns","#","doctest",":","+SKIP","[","Column","<","'","(","a",",","y",")","'",">",",","Column","<","'","(","b",",","z",")","'",">","]",">",">",">","internal.data_fields","#","doctest",":","+NORMALIZE_WHITESPACE","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","(","a",",","y",")","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","(","b",",","z",")","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.column_label_names","[","(","'column_labels_a","'",",",")",",","(","'column_labels_b","'",",",")","]"]
code: def __init__(
      self,
      spark_frame: PySparkDataFrame,
      index_spark_columns: Optional[List[PySparkColumn]],
      index_names: Optional[List[Optional[Label]]] = None,
      index_fields: Optional[List[InternalField]] = None,
      column_labels: Optional[List[Label]] = None,
      data_spark_columns: Optional[List[PySparkColumn]] = None,
      data_fields: Optional[List[InternalField]] = None,
      column_label_names: Optional[List[Optional[Label]]] = None,
  ):
      """
      Create a new internal immutable DataFrame to manage Spark DataFrame, column fields and
      index fields and names.
  
      :param spark_frame: Spark DataFrame to be managed.
      :param index_spark_columns: list of Spark Column
                                  Spark Columns for the index.
      :param index_names: list of tuples
                          the index names.
      :param index_fields: list of InternalField
                           the InternalFields for the index columns
      :param column_labels: list of tuples with the same length
                            The multi-level values in the tuples.
      :param data_spark_columns: list of Spark Column
                                 Spark Columns to appear as columns. If this is None, calculated
                                 from spark_frame.
      :param data_fields: list of InternalField
                          the InternalFields for the data columns
      :param column_label_names: Names for each of the column index levels.
  
      See the examples below to refer what each parameter means.
  
      >>> column_labels = pd.MultiIndex.from_tuples(
      ...     [('a', 'x'), ('a', 'y'), ('b', 'z')], names=["column_labels_a", "column_labels_b"])
      >>> row_index = pd.MultiIndex.from_tuples(
      ...     [('foo', 'bar'), ('foo', 'bar'), ('zoo', 'bar')],
      ...     names=["row_index_a", "row_index_b"])
      >>> psdf = ps.DataFrame(
      ...     [[1, 2, 3], [4, 5, 6], [7, 8, 9]], index=row_index, columns=column_labels)
      >>> psdf.set_index(('a', 'x'), append=True, inplace=True)
      >>> psdf  # doctest: +NORMALIZE_WHITESPACE
      column_labels_a                  a  b
      column_labels_b                  y  z
      row_index_a row_index_b (a, x)
      foo         bar         1       2  3
                              4       5  6
      zoo         bar         7       8  9
  
      >>> internal = psdf._internal
  
      >>> internal.spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
      +-----------------+-----------------+------+------+------+...
      |__index_level_0__|__index_level_1__|(a, x)|(a, y)|(b, z)|...
      +-----------------+-----------------+------+------+------+...
      |              foo|              bar|     1|     2|     3|...
      |              foo|              bar|     4|     5|     6|...
      |              zoo|              bar|     7|     8|     9|...
      +-----------------+-----------------+------+------+------+...
  
      >>> internal.index_spark_columns  # doctest: +SKIP
      [Column<'__index_level_0__'>, Column<'__index_level_1__'>, Column<'(a, x)'>]
  
      >>> internal.index_names
      [('row_index_a',), ('row_index_b',), ('a', 'x')]
  
      >>> internal.index_fields  # doctest: +NORMALIZE_WHITESPACE
      [InternalField(dtype=object,
          struct_field=StructField('__index_level_0__', StringType(), False)),
       InternalField(dtype=object,
          struct_field=StructField('__index_level_1__', StringType(), False)),
       InternalField(dtype=int64,
          struct_field=StructField('(a, x)', LongType(), False))]
  
      >>> internal.column_labels
      [('a', 'y'), ('b', 'z')]
  
      >>> internal.data_spark_columns  # doctest: +SKIP
      [Column<'(a, y)'>, Column<'(b, z)'>]
  
      >>> internal.data_fields  # doctest: +NORMALIZE_WHITESPACE
      [InternalField(dtype=int64, struct_field=StructField('(a, y)', LongType(), False)),
       InternalField(dtype=int64, struct_field=StructField('(b, z)', LongType(), False))]
  
      >>> internal.column_label_names
      [('column_labels_a',), ('column_labels_b',)]
      """
      SparkDataFrame = get_dataframe_class()
      assert isinstance(spark_frame, SparkDataFrame)
      assert (
          not spark_frame.isStreaming
      ), "pandas-on-Spark does not support Structured Streaming."
  
      if not index_spark_columns:
          if data_spark_columns is not None:
              if column_labels is not None:
                  data_spark_columns = [
                      scol.alias(name_like_string(label))
                      for scol, label in zip(
                          data_spark_columns, column_labels
                      )
                  ]
              spark_frame = spark_frame.select(data_spark_columns)
  
          assert not any(
              SPARK_INDEX_NAME_PATTERN.match(name)
              for name in spark_frame.columns
          ), (
              "Index columns should not appear in columns of the Spark DataFrame. Avoid "
              "index column names [%s]." % SPARK_INDEX_NAME_PATTERN
          )
  
          # Create default index.
          spark_frame = InternalFrame.attach_default_index(spark_frame)
          index_spark_columns = [
              scol_for(spark_frame, SPARK_DEFAULT_INDEX_NAME)
          ]
  
          index_fields = [
              InternalField.from_struct_field(
                  StructField(
                      SPARK_DEFAULT_INDEX_NAME,
                      LongType(),
                      nullable=False,
                  )
              )
          ]
  
          if data_spark_columns is not None:
              data_struct_fields = [
                  field
                  for field in spark_frame.schema.fields
                  if field.name != SPARK_DEFAULT_INDEX_NAME
              ]
              data_spark_columns = [
                  scol_for(spark_frame, field.name)
                  for field in data_struct_fields
              ]
              if data_fields is not None:
                  data_fields = [
                      field.copy(
                          name=name_like_string(struct_field.name),
                      )
                      for field, struct_field in zip(
                          data_fields, data_struct_fields
                      )
                  ]
  
      if NATURAL_ORDER_COLUMN_NAME not in spark_frame.columns:
          spark_frame = spark_frame.withColumn(
              NATURAL_ORDER_COLUMN_NAME, F.monotonically_increasing_id()
          )
  
      self._sdf = spark_frame
  
      # index_spark_columns
      Column = get_column_class()
      assert all(
          isinstance(index_scol, Column)
          for index_scol in index_spark_columns
      ), index_spark_columns
  
      self._index_spark_columns: List[Column] = index_spark_columns  # type: ignore[valid-type]
  
      # data_spark_columns
      if data_spark_columns is None:
          data_spark_columns = [
              scol_for(spark_frame, col)
              for col in spark_frame.columns
              if all(
                  not spark_column_equals(
                      scol_for(spark_frame, col), index_scol
                  )
                  for index_scol in index_spark_columns
              )
              and col not in HIDDEN_COLUMNS
          ]
      else:
          assert all(
              isinstance(scol, Column) for scol in data_spark_columns
          )
  
      self._data_spark_columns: List[Column] = data_spark_columns  # type: ignore[valid-type]
  
      # fields
      if index_fields is None:
          index_fields = [None] * len(index_spark_columns)
      if data_fields is None:
          data_fields = [None] * len(data_spark_columns)
  
      assert len(index_spark_columns) == len(index_fields), (
          len(index_spark_columns),
          len(index_fields),
      )
      assert len(data_spark_columns) == len(data_fields), (
          len(data_spark_columns),
          len(data_fields),
      )
  
      if any(
          field is None or field.struct_field is None
          for field in index_fields
      ) and any(
          field is None or field.struct_field is None
          for field in data_fields
      ):
          schema = spark_frame.select(
              index_spark_columns + data_spark_columns
          ).schema
          fields = [
              InternalField.from_struct_field(struct_field)
              if field is None
              else InternalField(field.dtype, struct_field)
              if field.struct_field is None
              else field
              for field, struct_field in zip(
                  index_fields + data_fields, schema.fields
              )
          ]
          index_fields = fields[: len(index_spark_columns)]
          data_fields = fields[len(index_spark_columns) :]
      elif any(
          field is None or field.struct_field is None
          for field in index_fields
      ):
          schema = spark_frame.select(index_spark_columns).schema
          index_fields = [
              InternalField.from_struct_field(struct_field)
              if field is None
              else InternalField(field.dtype, struct_field)
              if field.struct_field is None
              else field
              for field, struct_field in zip(
                  index_fields, schema.fields
              )
          ]
      elif any(
          field is None or field.struct_field is None
          for field in data_fields
      ):
          schema = spark_frame.select(data_spark_columns).schema
          data_fields = [
              InternalField.from_struct_field(struct_field)
              if field is None
              else InternalField(field.dtype, struct_field)
              if field.struct_field is None
              else field
              for field, struct_field in zip(data_fields, schema.fields)
          ]
  
      assert all(
          isinstance(ops.dtype, Dtype.__args__)  # type: ignore[attr-defined]
          and (
              ops.dtype == np.dtype("object")
              or as_spark_type(ops.dtype, raise_error=False) is not None
          )
          for ops in index_fields
      ), index_fields
  
      if is_testing():
          struct_fields = spark_frame.select(
              index_spark_columns
          ).schema.fields
          if is_remote():
              # TODO(SPARK-42965): For some reason, the metadata of StructField is different
              # in a few tests when using Spark Connect. However, the function works properly.
              # Therefore, we temporarily perform Spark Connect tests by excluding metadata
              # until the issue is resolved.
              def remove_metadata(
                  struct_field: StructField,
              ) -> StructField:
                  new_struct_field = StructField(
                      struct_field.name,
                      struct_field.dataType,
                      struct_field.nullable,
                  )
                  return new_struct_field
  
              assert all(
                  remove_metadata(index_field.struct_field)
                  == remove_metadata(struct_field)
                  for index_field, struct_field in zip(
                      index_fields, struct_fields
                  )
              ), (index_fields, struct_fields)
          else:
              assert all(
                  index_field.struct_field == struct_field
                  for index_field, struct_field in zip(
                      index_fields, struct_fields
                  )
              ), (index_fields, struct_fields)
  
      self._index_fields: List[InternalField] = index_fields
  
      assert all(
          isinstance(ops.dtype, Dtype.__args__)  # type: ignore[attr-defined]
          and (
              ops.dtype == np.dtype("object")
              or as_spark_type(ops.dtype, raise_error=False) is not None
          )
          for ops in data_fields
      ), data_fields
  
      if is_testing():
          struct_fields = spark_frame.select(
              data_spark_columns
          ).schema.fields
          if is_remote():
              # TODO(SPARK-42965): For some reason, the metadata of StructField is different
              # in a few tests when using Spark Connect. However, the function works properly.
              # Therefore, we temporarily perform Spark Connect tests by excluding metadata
              # until the issue is resolved.
              def remove_metadata(
                  struct_field: StructField,
              ) -> StructField:
                  new_struct_field = StructField(
                      struct_field.name,
                      struct_field.dataType,
                      struct_field.nullable,
                  )
                  return new_struct_field
  
              assert all(
                  remove_metadata(data_field.struct_field)
                  == remove_metadata(struct_field)
                  for data_field, struct_field in zip(
                      data_fields, struct_fields
                  )
              ), (data_fields, struct_fields)
          else:
              assert all(
                  data_field.struct_field == struct_field
                  for data_field, struct_field in zip(
                      data_fields, struct_fields
                  )
              ), (data_fields, struct_fields)
  
      self._data_fields: List[InternalField] = data_fields
  
      # index_names
      if not index_names:
          index_names = [None] * len(index_spark_columns)
  
      assert len(index_spark_columns) == len(index_names), (
          len(index_spark_columns),
          len(index_names),
      )
      assert all(
          is_name_like_tuple(index_name, check_type=True)
          for index_name in index_names
      ), index_names
  
      self._index_names: List[Optional[Label]] = index_names
  
      # column_labels
      if column_labels is None:
          column_labels = [
              (col,)
              for col in spark_frame.select(
                  self._data_spark_columns
              ).columns
          ]
      else:
          assert len(column_labels) == len(self._data_spark_columns), (
              len(column_labels),
              len(self._data_spark_columns),
          )
          if len(column_labels) == 1:
              column_label = column_labels[0]
              assert is_name_like_tuple(
                  column_label, check_type=True
              ), column_label
          else:
              assert all(
                  is_name_like_tuple(column_label, check_type=True)
                  for column_label in column_labels
              ), column_labels
              assert (
                  len(set(len(label) for label in column_labels)) <= 1
              ), column_labels
  
      self._column_labels: List[Label] = column_labels
  
      # column_label_names
      if column_label_names is None:
          column_label_names = [None] * column_labels_level(
              self._column_labels
          )
      else:
          if len(self._column_labels) > 0:
              assert len(column_label_names) == column_labels_level(
                  self._column_labels
              ), (
                  len(column_label_names),
                  column_labels_level(self._column_labels),
              )
          else:
              assert len(column_label_names) > 0, len(
                  column_label_names
              )
          assert all(
              is_name_like_tuple(column_label_name, check_type=True)
              for column_label_name in column_label_names
          ), column_label_names
  
      self._column_label_names: List[
          Optional[Label]
      ] = column_label_names
  
imports: import re
  from typing import Any, Dict, List, Optional, Sequence, Tuple, Union, TYPE_CHECKING, cast
  import numpy
  import pandas
  from pandas.api.types import CategoricalDtype
  from pyspark._globals import _NoValue, _NoValueType
  from pyspark.sql import functions, Column, DataFrame, Window
  from pyspark.sql.types import BooleanType, DataType, LongType, StructField, StructType, StringType
  from pyspark.sql.utils import is_timestamp_ntz_preferred
  from pyspark.sql.utils import is_remote, get_column_class, get_dataframe_class
  from pyspark import pandas
  from pyspark.pandas._typing import Label
  from pyspark.pandas.spark.utils import as_nullable_spark_type, force_decimal_precision_scale
  from pyspark.pandas.data_type_ops.base import DataTypeOps
  from pyspark.pandas.typedef import Dtype, as_spark_type, extension_dtypes, infer_pd_series_spark_type, spark_type_to_pandas_dtype
  from pyspark.pandas.utils import column_labels_level, default_session, is_name_like_tuple, is_testing, lazy_property, name_like_string, scol_for, spark_column_equals
label_desc: Use image node_id 1 to create a new InternalFrame object  with example: obj = InternalFrame(spark_frame, index_spark_columns, index_names, index_fields, column_labels, data_spark_columns, data_fields, column_label_names)