class_docstr: The internal immutable DataFrame which manages Spark DataFrame and column names and index
  information.
  
  .. note:: this is an internal class. It is not supposed to be exposed to users and users
      should not directly access to it.
  
  The internal immutable DataFrame represents the index information for a DataFrame it belongs to.
  For instance, if we have a pandas-on-Spark DataFrame as below, pandas DataFrame does not
  store the index as columns.
  
  >>> psdf = ps.DataFrame({
  ...     'A': [1, 2, 3, 4],
  ...     'B': [5, 6, 7, 8],
  ...     'C': [9, 10, 11, 12],
  ...     'D': [13, 14, 15, 16],
  ...     'E': [17, 18, 19, 20]}, columns = ['A', 'B', 'C', 'D', 'E'])
  >>> psdf  # doctest: +NORMALIZE_WHITESPACE
     A  B   C   D   E
  0  1  5   9  13  17
  1  2  6  10  14  18
  2  3  7  11  15  19
  3  4  8  12  16  20
  
  However, all columns including index column are also stored in Spark DataFrame internally
  as below.
  
  >>> psdf._internal.to_internal_spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE
  +-----------------+---+---+---+---+---+
  |__index_level_0__|  A|  B|  C|  D|  E|
  +-----------------+---+---+---+---+---+
  |                0|  1|  5|  9| 13| 17|
  |                1|  2|  6| 10| 14| 18|
  |                2|  3|  7| 11| 15| 19|
  |                3|  4|  8| 12| 16| 20|
  +-----------------+---+---+---+---+---+
  
  To fill this gap, the current metadata is used by mapping Spark's internal column
  to pandas-on-Spark's index. See the method below:
  
  * `spark_frame` represents the internal Spark DataFrame
  
  * `data_spark_column_names` represents non-indexing Spark column names
  
  * `data_spark_columns` represents non-indexing Spark columns
  
  * `data_fields` represents non-indexing InternalFields
  
  * `index_spark_column_names` represents internal index Spark column names
  
  * `index_spark_columns` represents internal index Spark columns
  
  * `index_fields` represents index InternalFields
  
  * `spark_column_names` represents all columns
  
  * `index_names` represents the external index name as a label
  
  * `to_internal_spark_frame` represents Spark DataFrame derived by the metadata. Includes index.
  
  * `to_pandas_frame` represents pandas DataFrame derived by the metadata
  
  >>> internal = psdf._internal
  >>> internal.spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
  +-----------------+---+---+---+---+---+-----------------+
  |__index_level_0__|  A|  B|  C|  D|  E|__natural_order__|
  +-----------------+---+---+---+---+---+-----------------+
  |                0|  1|  5|  9| 13| 17|              ...|
  |                1|  2|  6| 10| 14| 18|              ...|
  |                2|  3|  7| 11| 15| 19|              ...|
  |                3|  4|  8| 12| 16| 20|              ...|
  +-----------------+---+---+---+---+---+-----------------+
  >>> internal.data_spark_column_names
  ['A', 'B', 'C', 'D', 'E']
  >>> internal.index_spark_column_names
  ['__index_level_0__']
  >>> internal.spark_column_names
  ['__index_level_0__', 'A', 'B', 'C', 'D', 'E']
  >>> internal.index_names
  [None]
  >>> internal.data_fields    # doctest: +NORMALIZE_WHITESPACE
  [InternalField(dtype=int64, struct_field=StructField('A', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('B', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('C', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('D', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('E', LongType(), False))]
  >>> internal.index_fields
  [InternalField(dtype=int64, struct_field=StructField('__index_level_0__', LongType(), False))]
  >>> internal.to_internal_spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE
  +-----------------+---+---+---+---+---+
  |__index_level_0__|  A|  B|  C|  D|  E|
  +-----------------+---+---+---+---+---+
  |                0|  1|  5|  9| 13| 17|
  |                1|  2|  6| 10| 14| 18|
  |                2|  3|  7| 11| 15| 19|
  |                3|  4|  8| 12| 16| 20|
  +-----------------+---+---+---+---+---+
  >>> internal.to_pandas_frame
     A  B   C   D   E
  0  1  5   9  13  17
  1  2  6  10  14  18
  2  3  7  11  15  19
  3  4  8  12  16  20
  
  In case that index is set to one of the existing columns as below:
  
  >>> psdf1 = psdf.set_index("A")
  >>> psdf1  # doctest: +NORMALIZE_WHITESPACE
     B   C   D   E
  A
  1  5   9  13  17
  2  6  10  14  18
  3  7  11  15  19
  4  8  12  16  20
  
  >>> psdf1._internal.to_internal_spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE
  +---+---+---+---+---+
  |  A|  B|  C|  D|  E|
  +---+---+---+---+---+
  |  1|  5|  9| 13| 17|
  |  2|  6| 10| 14| 18|
  |  3|  7| 11| 15| 19|
  |  4|  8| 12| 16| 20|
  +---+---+---+---+---+
  
  >>> internal = psdf1._internal
  >>> internal.spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
  +-----------------+---+---+---+---+---+-----------------+
  |__index_level_0__|  A|  B|  C|  D|  E|__natural_order__|
  +-----------------+---+---+---+---+---+-----------------+
  |                0|  1|  5|  9| 13| 17|              ...|
  |                1|  2|  6| 10| 14| 18|              ...|
  |                2|  3|  7| 11| 15| 19|              ...|
  |                3|  4|  8| 12| 16| 20|              ...|
  +-----------------+---+---+---+---+---+-----------------+
  >>> internal.data_spark_column_names
  ['B', 'C', 'D', 'E']
  >>> internal.index_spark_column_names
  ['A']
  >>> internal.spark_column_names
  ['A', 'B', 'C', 'D', 'E']
  >>> internal.index_names
  [('A',)]
  >>> internal.data_fields  # doctest: +NORMALIZE_WHITESPACE
  [InternalField(dtype=int64, struct_field=StructField('B', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('C', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('D', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('E', LongType(), False))]
  >>> internal.index_fields
  [InternalField(dtype=int64, struct_field=StructField('A', LongType(), False))]
  >>> internal.to_internal_spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE
  +---+---+---+---+---+
  |  A|  B|  C|  D|  E|
  +---+---+---+---+---+
  |  1|  5|  9| 13| 17|
  |  2|  6| 10| 14| 18|
  |  3|  7| 11| 15| 19|
  |  4|  8| 12| 16| 20|
  +---+---+---+---+---+
  >>> internal.to_pandas_frame  # doctest: +NORMALIZE_WHITESPACE
     B   C   D   E
  A
  1  5   9  13  17
  2  6  10  14  18
  3  7  11  15  19
  4  8  12  16  20
  
  In case that index becomes a multi index as below:
  
  >>> psdf2 = psdf.set_index("A", append=True)
  >>> psdf2  # doctest: +NORMALIZE_WHITESPACE
       B   C   D   E
    A
  0 1  5   9  13  17
  1 2  6  10  14  18
  2 3  7  11  15  19
  3 4  8  12  16  20
  
  >>> psdf2._internal.to_internal_spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE
  +-----------------+---+---+---+---+---+
  |__index_level_0__|  A|  B|  C|  D|  E|
  +-----------------+---+---+---+---+---+
  |                0|  1|  5|  9| 13| 17|
  |                1|  2|  6| 10| 14| 18|
  |                2|  3|  7| 11| 15| 19|
  |                3|  4|  8| 12| 16| 20|
  +-----------------+---+---+---+---+---+
  
  >>> internal = psdf2._internal
  >>> internal.spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
  +-----------------+---+---+---+---+---+-----------------+
  |__index_level_0__|  A|  B|  C|  D|  E|__natural_order__|
  +-----------------+---+---+---+---+---+-----------------+
  |                0|  1|  5|  9| 13| 17|              ...|
  |                1|  2|  6| 10| 14| 18|              ...|
  |                2|  3|  7| 11| 15| 19|              ...|
  |                3|  4|  8| 12| 16| 20|              ...|
  +-----------------+---+---+---+---+---+-----------------+
  >>> internal.data_spark_column_names
  ['B', 'C', 'D', 'E']
  >>> internal.index_spark_column_names
  ['__index_level_0__', 'A']
  >>> internal.spark_column_names
  ['__index_level_0__', 'A', 'B', 'C', 'D', 'E']
  >>> internal.index_names
  [None, ('A',)]
  >>> internal.data_fields  # doctest: +NORMALIZE_WHITESPACE
  [InternalField(dtype=int64, struct_field=StructField('B', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('C', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('D', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('E', LongType(), False))]
  >>> internal.index_fields  # doctest: +NORMALIZE_WHITESPACE
  [InternalField(dtype=int64, struct_field=StructField('__index_level_0__', LongType(), False)),
   InternalField(dtype=int64, struct_field=StructField('A', LongType(), False))]
  >>> internal.to_internal_spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE
  +-----------------+---+---+---+---+---+
  |__index_level_0__|  A|  B|  C|  D|  E|
  +-----------------+---+---+---+---+---+
  |                0|  1|  5|  9| 13| 17|
  |                1|  2|  6| 10| 14| 18|
  |                2|  3|  7| 11| 15| 19|
  |                3|  4|  8| 12| 16| 20|
  +-----------------+---+---+---+---+---+
  >>> internal.to_pandas_frame  # doctest: +NORMALIZE_WHITESPACE
       B   C   D   E
    A
  0 1  5   9  13  17
  1 2  6  10  14  18
  2 3  7  11  15  19
  3 4  8  12  16  20
  
  For multi-level columns, it also holds column_labels
  
  >>> columns = pd.MultiIndex.from_tuples([('X', 'A'), ('X', 'B'),
  ...                                      ('Y', 'C'), ('Y', 'D')])
  >>> psdf3 = ps.DataFrame([
  ...     [1, 2, 3, 4],
  ...     [5, 6, 7, 8],
  ...     [9, 10, 11, 12],
  ...     [13, 14, 15, 16],
  ...     [17, 18, 19, 20]], columns = columns)
  >>> psdf3  # doctest: +NORMALIZE_WHITESPACE
      X       Y
      A   B   C   D
  0   1   2   3   4
  1   5   6   7   8
  2   9  10  11  12
  3  13  14  15  16
  4  17  18  19  20
  
  >>> internal = psdf3._internal
  >>> internal.spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
  +-----------------+------+------+------+------+-----------------+
  |__index_level_0__|(X, A)|(X, B)|(Y, C)|(Y, D)|__natural_order__|
  +-----------------+------+------+------+------+-----------------+
  |                0|     1|     2|     3|     4|              ...|
  |                1|     5|     6|     7|     8|              ...|
  |                2|     9|    10|    11|    12|              ...|
  |                3|    13|    14|    15|    16|              ...|
  |                4|    17|    18|    19|    20|              ...|
  +-----------------+------+------+------+------+-----------------+
  >>> internal.data_spark_column_names
  ['(X, A)', '(X, B)', '(Y, C)', '(Y, D)']
  >>> internal.column_labels
  [('X', 'A'), ('X', 'B'), ('Y', 'C'), ('Y', 'D')]
  
  For Series, it also holds scol to represent the column.
  
  >>> psseries = psdf1.B
  >>> psseries
  A
  1    5
  2    6
  3    7
  4    8
  Name: B, dtype: int64
  
  >>> internal = psseries._internal
  >>> internal.spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
  +-----------------+---+---+---+---+---+-----------------+
  |__index_level_0__|  A|  B|  C|  D|  E|__natural_order__|
  +-----------------+---+---+---+---+---+-----------------+
  |                0|  1|  5|  9| 13| 17|              ...|
  |                1|  2|  6| 10| 14| 18|              ...|
  |                2|  3|  7| 11| 15| 19|              ...|
  |                3|  4|  8| 12| 16| 20|              ...|
  +-----------------+---+---+---+---+---+-----------------+
  >>> internal.data_spark_column_names
  ['B']
  >>> internal.index_spark_column_names
  ['A']
  >>> internal.spark_column_names
  ['A', 'B']
  >>> internal.index_names
  [('A',)]
  >>> internal.data_fields
  [InternalField(dtype=int64, struct_field=StructField('B', LongType(), False))]
  >>> internal.index_fields
  [InternalField(dtype=int64, struct_field=StructField('A', LongType(), False))]
  >>> internal.to_internal_spark_frame.show()  # doctest: +NORMALIZE_WHITESPACE
  +---+---+
  |  A|  B|
  +---+---+
  |  1|  5|
  |  2|  6|
  |  3|  7|
  |  4|  8|
  +---+---+
  >>> internal.to_pandas_frame  # doctest: +NORMALIZE_WHITESPACE
     B
  A
  1  5
  2  6
  3  7
  4  8
class_docstr_tok: ["The","internal","immutable","DataFrame","which","manages","Spark","DataFrame","and","column","names","and","index","information",".","..","note",":",":","this","is","an","internal","class",".","It","is","not","supposed","to","be","exposed","to","users","and","users","should","not","directly","access","to","it",".","The","internal","immutable","DataFrame","represents","the","index","information","for","a","DataFrame","it","belongs","to",".","For","instance",",","if","we","have","a","pandas-on-Spark","DataFrame","as","below",",","pandas","DataFrame","does","not","store","the","index","as","columns",".",">",">",">","psdf","=","ps.DataFrame","(","{","...","'","A","'",":","[","1",",","2",",","3",",","4","]",",","...","'","B","'",":","[","5",",","6",",","7",",","8","]",",","...","'","C","'",":","[","9",",","10",",","11",",","12","]",",","...","'D","'",":","[","13",",","14",",","15",",","16","]",",","...","'","E","'",":","[","17",",","18",",","19",",","20","]","}",",","columns","=","[","'","A","'",",","'","B","'",",","'","C","'",",","'D","'",",","'","E","'","]",")",">",">",">","psdf","#","doctest",":","+NORMALIZE_WHITESPACE","A","B","C","D","E","0","1","5","9","13","17","1","2","6","10","14","18","2","3","7","11","15","19","3","4","8","12","16","20","However",",","all","columns","including","index","column","are","also","stored","in","Spark","DataFrame","internally","as","below",".",">",">",">","psdf._internal.to_internal_spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","|__index_level_0__|","A|","B|","C|","D|","E|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","|","0|","1|","5|","9|","13|","17|","|","1|","2|","6|","10|","14|","18|","|","2|","3|","7|","11|","15|","19|","|","3|","4|","8|","12|","16|","20|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","To","fill","this","gap",",","the","current","metadata","is","used","by","mapping","Spark","'s","internal","column","to","pandas-on-Spark","'s","index",".","See","the","method","below",":","*","`","spark_frame","`","represents","the","internal","Spark","DataFrame","*","`","data_spark_column_names","`","represents","non-indexing","Spark","column","names","*","`","data_spark_columns","`","represents","non-indexing","Spark","columns","*","`","data_fields","`","represents","non-indexing","InternalFields","*","`","index_spark_column_names","`","represents","internal","index","Spark","column","names","*","`","index_spark_columns","`","represents","internal","index","Spark","columns","*","`","index_fields","`","represents","index","InternalFields","*","`","spark_column_names","`","represents","all","columns","*","`","index_names","`","represents","the","external","index","name","as","a","label","*","`","to_internal_spark_frame","`","represents","Spark","DataFrame","derived","by","the","metadata",".","Includes","index",".","*","`","to_pandas_frame","`","represents","pandas","DataFrame","derived","by","the","metadata",">",">",">","internal","=","psdf._internal",">",">",">","internal.spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+ELLIPSIS","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+","|__index_level_0__|","A|","B|","C|","D|","E|__natural_order__|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+","|","0|","1|","5|","9|","13|","17|","...","|","|","1|","2|","6|","10|","14|","18|","...","|","|","2|","3|","7|","11|","15|","19|","...","|","|","3|","4|","8|","12|","16|","20|","...","|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+",">",">",">","internal.data_spark_column_names","[","'","A","'",",","'","B","'",",","'","C","'",",","'D","'",",","'","E","'","]",">",">",">","internal.index_spark_column_names","[","'__index_level_0__","'","]",">",">",">","internal.spark_column_names","[","'__index_level_0__","'",",","'","A","'",",","'","B","'",",","'","C","'",",","'D","'",",","'","E","'","]",">",">",">","internal.index_names","[","None","]",">",">",">","internal.data_fields","#","doctest",":","+NORMALIZE_WHITESPACE","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","A","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","B","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","C","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'D","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","E","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.index_fields","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'__index_level_0__","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.to_internal_spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","|__index_level_0__|","A|","B|","C|","D|","E|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","|","0|","1|","5|","9|","13|","17|","|","1|","2|","6|","10|","14|","18|","|","2|","3|","7|","11|","15|","19|","|","3|","4|","8|","12|","16|","20|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+",">",">",">","internal.to_pandas_frame","A","B","C","D","E","0","1","5","9","13","17","1","2","6","10","14","18","2","3","7","11","15","19","3","4","8","12","16","20","In","case","that","index","is","set","to","one","of","the","existing","columns","as","below",":",">",">",">","psdf1","=","psdf.set_index","(","``","A","''",")",">",">",">","psdf1","#","doctest",":","+NORMALIZE_WHITESPACE","B","C","D","E","A","1","5","9","13","17","2","6","10","14","18","3","7","11","15","19","4","8","12","16","20",">",">",">","psdf1._internal.to_internal_spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+","--","-+","--","-+","--","-+","--","-+","--","-+","|","A|","B|","C|","D|","E|","+","--","-+","--","-+","--","-+","--","-+","--","-+","|","1|","5|","9|","13|","17|","|","2|","6|","10|","14|","18|","|","3|","7|","11|","15|","19|","|","4|","8|","12|","16|","20|","+","--","-+","--","-+","--","-+","--","-+","--","-+",">",">",">","internal","=","psdf1._internal",">",">",">","internal.spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+ELLIPSIS","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+","|__index_level_0__|","A|","B|","C|","D|","E|__natural_order__|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+","|","0|","1|","5|","9|","13|","17|","...","|","|","1|","2|","6|","10|","14|","18|","...","|","|","2|","3|","7|","11|","15|","19|","...","|","|","3|","4|","8|","12|","16|","20|","...","|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+",">",">",">","internal.data_spark_column_names","[","'","B","'",",","'","C","'",",","'D","'",",","'","E","'","]",">",">",">","internal.index_spark_column_names","[","'","A","'","]",">",">",">","internal.spark_column_names","[","'","A","'",",","'","B","'",",","'","C","'",",","'D","'",",","'","E","'","]",">",">",">","internal.index_names","[","(","'","A","'",",",")","]",">",">",">","internal.data_fields","#","doctest",":","+NORMALIZE_WHITESPACE","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","B","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","C","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'D","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","E","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.index_fields","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","A","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.to_internal_spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+","--","-+","--","-+","--","-+","--","-+","--","-+","|","A|","B|","C|","D|","E|","+","--","-+","--","-+","--","-+","--","-+","--","-+","|","1|","5|","9|","13|","17|","|","2|","6|","10|","14|","18|","|","3|","7|","11|","15|","19|","|","4|","8|","12|","16|","20|","+","--","-+","--","-+","--","-+","--","-+","--","-+",">",">",">","internal.to_pandas_frame","#","doctest",":","+NORMALIZE_WHITESPACE","B","C","D","E","A","1","5","9","13","17","2","6","10","14","18","3","7","11","15","19","4","8","12","16","20","In","case","that","index","becomes","a","multi","index","as","below",":",">",">",">","psdf2","=","psdf.set_index","(","``","A","''",",","append=True",")",">",">",">","psdf2","#","doctest",":","+NORMALIZE_WHITESPACE","B","C","D","E","A","0","1","5","9","13","17","1","2","6","10","14","18","2","3","7","11","15","19","3","4","8","12","16","20",">",">",">","psdf2._internal.to_internal_spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","|__index_level_0__|","A|","B|","C|","D|","E|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","|","0|","1|","5|","9|","13|","17|","|","1|","2|","6|","10|","14|","18|","|","2|","3|","7|","11|","15|","19|","|","3|","4|","8|","12|","16|","20|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+",">",">",">","internal","=","psdf2._internal",">",">",">","internal.spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+ELLIPSIS","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+","|__index_level_0__|","A|","B|","C|","D|","E|__natural_order__|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+","|","0|","1|","5|","9|","13|","17|","...","|","|","1|","2|","6|","10|","14|","18|","...","|","|","2|","3|","7|","11|","15|","19|","...","|","|","3|","4|","8|","12|","16|","20|","...","|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+",">",">",">","internal.data_spark_column_names","[","'","B","'",",","'","C","'",",","'D","'",",","'","E","'","]",">",">",">","internal.index_spark_column_names","[","'__index_level_0__","'",",","'","A","'","]",">",">",">","internal.spark_column_names","[","'__index_level_0__","'",",","'","A","'",",","'","B","'",",","'","C","'",",","'D","'",",","'","E","'","]",">",">",">","internal.index_names","[","None",",","(","'","A","'",",",")","]",">",">",">","internal.data_fields","#","doctest",":","+NORMALIZE_WHITESPACE","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","B","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","C","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'D","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","E","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.index_fields","#","doctest",":","+NORMALIZE_WHITESPACE","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'__index_level_0__","'",",","LongType","(",")",",","False",")",")",",","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","A","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.to_internal_spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","|__index_level_0__|","A|","B|","C|","D|","E|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","|","0|","1|","5|","9|","13|","17|","|","1|","2|","6|","10|","14|","18|","|","2|","3|","7|","11|","15|","19|","|","3|","4|","8|","12|","16|","20|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+",">",">",">","internal.to_pandas_frame","#","doctest",":","+NORMALIZE_WHITESPACE","B","C","D","E","A","0","1","5","9","13","17","1","2","6","10","14","18","2","3","7","11","15","19","3","4","8","12","16","20","For","multi-level","columns",",","it","also","holds","column_labels",">",">",">","columns","=","pd.MultiIndex.from_tuples","(","[","(","'","X","'",",","'","A","'",")",",","(","'","X","'",",","'","B","'",")",",","...","(","'","Y","'",",","'","C","'",")",",","(","'","Y","'",",","'D","'",")","]",")",">",">",">","psdf3","=","ps.DataFrame","(","[","...","[","1",",","2",",","3",",","4","]",",","...","[","5",",","6",",","7",",","8","]",",","...","[","9",",","10",",","11",",","12","]",",","...","[","13",",","14",",","15",",","16","]",",","...","[","17",",","18",",","19",",","20","]","]",",","columns","=","columns",")",">",">",">","psdf3","#","doctest",":","+NORMALIZE_WHITESPACE","X","Y","A","B","C","D","0","1","2","3","4","1","5","6","7","8","2","9","10","11","12","3","13","14","15","16","4","17","18","19","20",">",">",">","internal","=","psdf3._internal",">",">",">","internal.spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+ELLIPSIS","+","--","--","--","--","--","--","--","--","-+","--","--","--","+","--","--","--","+","--","--","--","+","--","--","--","+","--","--","--","--","--","--","--","--","-+","|__index_level_0__|","(","X",",","A",")","|","(","X",",","B",")","|","(","Y",",","C",")","|","(","Y",",","D",")","|__natural_order__|","+","--","--","--","--","--","--","--","--","-+","--","--","--","+","--","--","--","+","--","--","--","+","--","--","--","+","--","--","--","--","--","--","--","--","-+","|","0|","1|","2|","3|","4|","...","|","|","1|","5|","6|","7|","8|","...","|","|","2|","9|","10|","11|","12|","...","|","|","3|","13|","14|","15|","16|","...","|","|","4|","17|","18|","19|","20|","...","|","+","--","--","--","--","--","--","--","--","-+","--","--","--","+","--","--","--","+","--","--","--","+","--","--","--","+","--","--","--","--","--","--","--","--","-+",">",">",">","internal.data_spark_column_names","[","'","(","X",",","A",")","'",",","'","(","X",",","B",")","'",",","'","(","Y",",","C",")","'",",","'","(","Y",",","D",")","'","]",">",">",">","internal.column_labels","[","(","'","X","'",",","'","A","'",")",",","(","'","X","'",",","'","B","'",")",",","(","'","Y","'",",","'","C","'",")",",","(","'","Y","'",",","'D","'",")","]","For","Series",",","it","also","holds","scol","to","represent","the","column",".",">",">",">","psseries","=","psdf1.B",">",">",">","psseries","A","1","5","2","6","3","7","4","8","Name",":","B",",","dtype",":","int64",">",">",">","internal","=","psseries._internal",">",">",">","internal.spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+ELLIPSIS","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+","|__index_level_0__|","A|","B|","C|","D|","E|__natural_order__|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+","|","0|","1|","5|","9|","13|","17|","...","|","|","1|","2|","6|","10|","14|","18|","...","|","|","2|","3|","7|","11|","15|","19|","...","|","|","3|","4|","8|","12|","16|","20|","...","|","+","--","--","--","--","--","--","--","--","-+","--","-+","--","-+","--","-+","--","-+","--","-+","--","--","--","--","--","--","--","--","-+",">",">",">","internal.data_spark_column_names","[","'","B","'","]",">",">",">","internal.index_spark_column_names","[","'","A","'","]",">",">",">","internal.spark_column_names","[","'","A","'",",","'","B","'","]",">",">",">","internal.index_names","[","(","'","A","'",",",")","]",">",">",">","internal.data_fields","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","B","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.index_fields","[","InternalField","(","dtype=int64",",","struct_field=StructField","(","'","A","'",",","LongType","(",")",",","False",")",")","]",">",">",">","internal.to_internal_spark_frame.show","(",")","#","doctest",":","+NORMALIZE_WHITESPACE","+","--","-+","--","-+","|","A|","B|","+","--","-+","--","-+","|","1|","5|","|","2|","6|","|","3|","7|","|","4|","8|","+","--","-+","--","-+",">",">",">","internal.to_pandas_frame","#","doctest",":","+NORMALIZE_WHITESPACE","B","A","1","5","2","6","3","7","4","8"]
docstr: Restore pandas DataFrame indices using the metadata.
  
  :param pdf: the pandas DataFrame to be processed.
  :param index_columns: the original column names for index columns.
  :param index_names: the index names after restored.
  :param data_columns: the original column names for data columns.
  :param column_labels: the column labels after restored.
  :param column_label_names: the column label names after restored.
  :param fields: the fields after restored.
  :return: the restored pandas DataFrame
  
  >>> from numpy import dtype
  >>> pdf = pd.DataFrame({"index": [10, 20, 30], "a": ['a', 'b', 'c'], "b": [0, 2, 1]})
  >>> InternalFrame.restore_index(
  ...     pdf,
  ...     index_columns=["index"],
  ...     index_names=[("idx",)],
  ...     data_columns=["a", "b", "index"],
  ...     column_labels=[("x",), ("y",), ("z",)],
  ...     column_label_names=[("lv1",)],
  ...     fields=[
  ...         InternalField(
  ...             dtype=dtype('int64'),
  ...             struct_field=StructField(name='index', dataType=LongType(), nullable=False),
  ...         ),
  ...         InternalField(
  ...             dtype=dtype('object'),
  ...             struct_field=StructField(name='a', dataType=StringType(), nullable=False),
  ...         ),
  ...         InternalField(
  ...             dtype=CategoricalDtype(categories=["i", "j", "k"]),
  ...             struct_field=StructField(name='b', dataType=LongType(), nullable=False),
  ...         ),
  ...     ],
  ... )  # doctest: +NORMALIZE_WHITESPACE
  lv1  x  y   z
  idx
  10   a  i  10
  20   b  k  20
  30   c  j  30
docstr_tok: ["Restore","pandas","DataFrame","indices","using","the","metadata",".",":","param","pdf",":","the","pandas","DataFrame","to","be","processed",".",":","param","index_columns",":","the","original","column","names","for","index","columns",".",":","param","index_names",":","the","index","names","after","restored",".",":","param","data_columns",":","the","original","column","names","for","data","columns",".",":","param","column_labels",":","the","column","labels","after","restored",".",":","param","column_label_names",":","the","column","label","names","after","restored",".",":","param","fields",":","the","fields","after","restored",".",":","return",":","the","restored","pandas","DataFrame",">",">",">","from","numpy","import","dtype",">",">",">","pdf","=","pd.DataFrame","(","{","``","index","''",":","[","10",",","20",",","30","]",",","``","a","''",":","[","'","a","'",",","'","b","'",",","'","c","'","]",",","``","b","''",":","[","0",",","2",",","1","]","}",")",">",">",">","InternalFrame.restore_index","(","...","pdf",",","...","index_columns=","[","``","index","''","]",",","...","index_names=","[","(","``","idx","''",",",")","]",",","...","data_columns=","[","``","a","''",",","``","b","''",",","``","index","''","]",",","...","column_labels=","[","(","``","x","''",",",")",",","(","``","y","''",",",")",",","(","``","z","''",",",")","]",",","...","column_label_names=","[","(","``","lv1","''",",",")","]",",","...","fields=","[","...","InternalField","(","...","dtype=dtype","(","'int64","'",")",",","...","struct_field=StructField","(","name='index","'",",","dataType=LongType","(",")",",","nullable=False",")",",","...",")",",","...","InternalField","(","...","dtype=dtype","(","'object","'",")",",","...","struct_field=StructField","(","name=","'","a","'",",","dataType=StringType","(",")",",","nullable=False",")",",","...",")",",","...","InternalField","(","...","dtype=CategoricalDtype","(","categories=","[","``","i","''",",","``","j","''",",","``","k","''","]",")",",","...","struct_field=StructField","(","name=","'","b","'",",","dataType=LongType","(",")",",","nullable=False",")",",","...",")",",","...","]",",","...",")","#","doctest",":","+NORMALIZE_WHITESPACE","lv1","x","y","z","idx","10","a","i","10","20","b","k","20","30","c","j","30"]
code: def restore_index(
      pdf: pd.DataFrame,
      *,
      index_columns: List[str],
      index_names: List[Label],
      data_columns: List[str],
      column_labels: List[Label],
      column_label_names: List[Label],
      fields: List[InternalField] = None,
  ) -> pd.DataFrame:
      """
      Restore pandas DataFrame indices using the metadata.
  
      :param pdf: the pandas DataFrame to be processed.
      :param index_columns: the original column names for index columns.
      :param index_names: the index names after restored.
      :param data_columns: the original column names for data columns.
      :param column_labels: the column labels after restored.
      :param column_label_names: the column label names after restored.
      :param fields: the fields after restored.
      :return: the restored pandas DataFrame
  
      >>> from numpy import dtype
      >>> pdf = pd.DataFrame({"index": [10, 20, 30], "a": ['a', 'b', 'c'], "b": [0, 2, 1]})
      >>> InternalFrame.restore_index(
      ...     pdf,
      ...     index_columns=["index"],
      ...     index_names=[("idx",)],
      ...     data_columns=["a", "b", "index"],
      ...     column_labels=[("x",), ("y",), ("z",)],
      ...     column_label_names=[("lv1",)],
      ...     fields=[
      ...         InternalField(
      ...             dtype=dtype('int64'),
      ...             struct_field=StructField(name='index', dataType=LongType(), nullable=False),
      ...         ),
      ...         InternalField(
      ...             dtype=dtype('object'),
      ...             struct_field=StructField(name='a', dataType=StringType(), nullable=False),
      ...         ),
      ...         InternalField(
      ...             dtype=CategoricalDtype(categories=["i", "j", "k"]),
      ...             struct_field=StructField(name='b', dataType=LongType(), nullable=False),
      ...         ),
      ...     ],
      ... )  # doctest: +NORMALIZE_WHITESPACE
      lv1  x  y   z
      idx
      10   a  i  10
      20   b  k  20
      30   c  j  30
      """
      for col, field in zip(pdf.columns, fields):
          pdf[col] = DataTypeOps(field.dtype, field.spark_type).restore(
              pdf[col]
          )
  
      append = False
      for index_field in index_columns:
          drop = index_field not in data_columns
          pdf = pdf.set_index(index_field, drop=drop, append=append)
          append = True
      pdf = pdf[data_columns]
  
      pdf.index.names = [
          name if name is None or len(name) > 1 else name[0]
          for name in index_names
      ]
  
      names = [
          name if name is None or len(name) > 1 else name[0]
          for name in column_label_names
      ]
      if len(column_label_names) > 1:
          pdf.columns = pd.MultiIndex.from_tuples(
              column_labels, names=names
          )
      else:
          pdf.columns = pd.Index(
              [
                  None if label is None else label[0]
                  for label in column_labels
              ],
              name=names[0],
          )
  
      return pdf
  
imports: import re
  from typing import Any, Dict, List, Optional, Sequence, Tuple, Union, TYPE_CHECKING, cast
  import numpy
  import pandas
  from pandas.api.types import CategoricalDtype
  from pyspark._globals import _NoValue, _NoValueType
  from pyspark.sql import functions, Column, DataFrame, Window
  from pyspark.sql.types import BooleanType, DataType, LongType, StructField, StructType, StringType
  from pyspark.sql.utils import is_timestamp_ntz_preferred
  from pyspark.sql.utils import is_remote, get_column_class, get_dataframe_class
  from pyspark import pandas
  from pyspark.pandas._typing import Label
  from pyspark.pandas.spark.utils import as_nullable_spark_type, force_decimal_precision_scale
  from pyspark.pandas.data_type_ops.base import DataTypeOps
  from pyspark.pandas.typedef import Dtype, as_spark_type, extension_dtypes, infer_pd_series_spark_type, spark_type_to_pandas_dtype
  from pyspark.pandas.utils import column_labels_level, default_session, is_name_like_tuple, is_testing, lazy_property, name_like_string, scol_for, spark_column_equals
label_desc: Use image node_id 28 for calling the InternalFrame obj's underlying member method code with example usage: obj.restore_index(pdf) and returns: pdf