class_docstr: A language model on English Wikipedia.
  
  References and internal links are removed from the raw XML.
  
  Special pages (non-articles) are dropped.
  
  This more closely resembles plain text, though there are still some xml
  elements, like tables.
  
  Each article is prefixed by a line containing the title and length in
  characters - e.g.
  title: "Price of Tea in China" length: 12345
  During inference time, you can forward generate starting with such a header
  in order to obtain a randomly generated article with a given title and
  (approximate) length.
  
  Result is chopped arbitrarily into sequences of length 1024 tokens,
  without regard to article boundaries.
class_docstr_tok: ["A","language","model","on","English","Wikipedia",".","References","and","internal","links","are","removed","from","the","raw","XML",".","Special","pages","(","non-articles",")","are","dropped",".","This","more","closely","resembles","plain","text",",","though","there","are","still","some","xml","elements",",","like","tables",".","Each","article","is","prefixed","by","a","line","containing","the","title","and","length","in","characters","-","e.g",".","title",":","``","Price","of","Tea","in","China","''","length",":","12345","During","inference","time",",","you","can","forward","generate","starting","with","such","a","header","in","order","to","obtain","a","randomly","generated","article","with","a","given","title","and","(","approximate",")","length",".","Result","is","chopped","arbitrarily","into","sequences","of","length","1024","tokens",",","without","regard","to","article","boundaries","."]
docstr: Overrides the base class to clean up the xml dump before tokenizing.
docstr_tok: ["Overrides","the","base","class","to","clean","up","the","xml","dump","before","tokenizing","."]
code: def filepath_to_unicode_strings(self, filepath):
      """Overrides the base class to clean up the xml dump before tokenizing."""
      dump = text_encoder.to_unicode_ignore_errors(
          tf.gfile.Open(filepath).read()
      )
      pages = _dump_to_pages(dump)
      ret = ""
      for p in pages:
          title = _page_to_title(p)
          text = _page_to_text(p)
          text = _remove_triple_quotes(
              _remove_double_brackets(_remove_references(text))
          )
          if ":" in title:
              # not a regular article
              continue
          if len(text) <= 140:
              # Probably a redirect or something like that.  Skip it.
              continue
          ret += 'title: "%s" length: %d\n%s\n' % (
              title,
              len(text),
              text,
          )
      yield ret
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import os
  import subprocess
  import numpy
  from tensor2tensor.data_generators import generator_utils
  from tensor2tensor.data_generators import problem
  from tensor2tensor.data_generators import text_encoder
  from tensor2tensor.data_generators import text_problems
  from tensor2tensor.utils import registry
  import tensorflow.compat.v1
label_desc: Use image node_id 1 for calling the LanguagemodelWikiNorefV8kL1k obj's underlying member method code with example usage: obj.filepath_to_unicode_strings(filepath) without return types