class_docstr: Transform one or more tokenized documents into a sparse document-term matrix
  of shape (# docs, # unique terms), with flexible weighting/normalization of values.
  
  Stream a corpus with metadata from disk:
  
  .. code-block:: pycon
  
      >>> ds = textacy.datasets.CapitolWords()
      >>> records = ds.records(limit=1000)
      >>> corpus = textacy.Corpus("en_core_web_sm", data=records)
      >>> print(corpus)
      Corpus(1000 docs, 538397 tokens)
  
  Tokenize and vectorize the first 600 documents of this corpus:
  
  .. code-block:: pycon
  
      >>> tokenized_docs = (
      ...     (term.lemma_ for term in textacy.extract.terms(doc, ngs=1, ents=True))
      ...     for doc in corpus[:600])
      >>> vectorizer = Vectorizer(
      ...     tf_type="linear", idf_type="smooth", norm="l2",
      ...     min_df=3, max_df=0.95)
      >>> doc_term_matrix = vectorizer.fit_transform(tokenized_docs)
      >>> doc_term_matrix
      <600x4412 sparse matrix of type '<class 'numpy.float64'>'
              with 65210 stored elements in Compressed Sparse Row format>
  
  Tokenize and vectorize the remaining 400 documents of the corpus, using only
  the groups, terms, and weights learned in the previous step:
  
  .. code-block:: pycon
  
      >>> tokenized_docs = (
      ...     (term.lemma_ for term in textacy.extract.terms(doc, ngs=1, ents=True))
      ...     for doc in corpus[600:])
      >>> doc_term_matrix = vectorizer.transform(tokenized_docs)
      >>> doc_term_matrix
      <400x4412 sparse matrix of type '<class 'numpy.float64'>'
              with 36212 stored elements in Compressed Sparse Row format>
  
  Inspect the terms associated with columns; they're sorted alphabetically:
  
  .. code-block:: pycon
  
      >>> vectorizer.terms_list[:5]
      ['', '$', '$ 1 million', '$ 1.2 billion', '$ 10 billion']
  
  (Btw: That empty string shouldn't be there. Somehow, spaCy is labeling it as
  a named entity...?)
  
  If known in advance, limit the terms included in vectorized outputs
  to a particular set of values:
  
  .. code-block:: pycon
  
      >>> tokenized_docs = (
      ...     (term.lemma_ for term in textacy.extract.terms(doc, ngs=1, ents=True))
      ...     for doc in corpus[:600])
      >>> vectorizer = Vectorizer(
      ...     idf_type="smooth", norm="l2",
      ...     min_df=3, max_df=0.95,
      ...     vocabulary_terms=["president", "bill", "unanimous", "distinguished", "american"])
      >>> doc_term_matrix = vectorizer.fit_transform(tokenized_docs)
      >>> doc_term_matrix
      <600x5 sparse matrix of type '<class 'numpy.float64'>'
              with 516 stored elements in Compressed Sparse Row format>
      >>> vectorizer.terms_list
      ['american', 'bill', 'distinguished', 'president', 'unanimous']
  
  Specify different weighting schemes to determine values in the matrix,
  adding or customizing individual components, as desired:
  
  .. code-block:: pycon
  
      >>> tokenized_docs = [
          [term.lemma_ for term in textacy.extract.terms(doc, ngs=1, ents=True)]
          for doc in corpus[:600]]
      >>> doc_term_matrix = Vectorizer(
      ...     tf_type="linear", norm=None, min_df=3, max_df=0.95
      ... ).fit_transform(tokenized_docs)
      >>> print(doc_term_matrix[:8, vectorizer.vocabulary_terms["$"]].toarray())
      [[0]
       [0]
       [1]
       [4]
       [0]
       [0]
       [2]
       [4]]
      >>> doc_term_matrix = Vectorizer(
      ...     tf_type="sqrt", dl_type="sqrt", norm=None, min_df=3, max_df=0.95
      ... ).fit_transform(tokenized_docs)
      >>> print(doc_term_matrix[:8, vectorizer.vocabulary_terms["$"]].toarray())
      [[0.        ]
       [0.        ]
       [0.10660036]
       [0.2773501 ]
       [0.        ]
       [0.        ]
       [0.11704115]
       [0.24806947]]
      >>> doc_term_matrix = Vectorizer(
      ...     tf_type="bm25", idf_type="smooth", norm=None, min_df=3, max_df=0.95
      ... ).fit_transform(tokenized_docs)
      >>> print(doc_term_matrix[:8, vectorizer.vocabulary_terms["$"]].toarray())
      [[0.        ]
       [0.        ]
       [2.68009606]
       [4.97732126]
       [0.        ]
       [0.        ]
       [3.87124987]
       [4.97732126]]
  
  If you're not sure what's going on mathematically, :attr:`Vectorizer.weighting`
  gives the formula being used to calculate weights, based on the parameters
  set when initializing the vectorizer:
  
  .. code-block:: pycon
  
      >>> vectorizer.weighting
      '(tf * (k + 1)) / (k + tf) * log((n_docs + 1) / (df + 1)) + 1'
  
  In general, weights may consist of a local component (term frequency),
  a global component (inverse document frequency), and a normalization
  component (document length). Individual components may be modified:
  they may have different scaling (e.g. tf vs. sqrt(tf)) or different behaviors
  (e.g. "standard" idf vs bm25's version). There are *many* possible weightings,
  and some may be better for particular use cases than others. When in doubt,
  though, just go with something standard.
  
  - "tf": Weights are simply the absolute per-document term frequencies (tfs),
    i.e. value (i, j) in an output doc-term matrix corresponds to the number
    of occurrences of term j in doc i. Terms appearing many times in a given
    doc receive higher weights than less common terms.
    Params: ``tf_type="linear", apply_idf=False, apply_dl=False``
  - "tfidf": Doc-specific, *local* tfs are multiplied by their corpus-wide,
    *global* inverse document frequencies (idfs). Terms appearing in many docs
    have higher document frequencies (dfs), correspondingly smaller idfs, and
    in turn, lower weights.
    Params: ``tf_type="linear", apply_idf=True, idf_type="smooth", apply_dl=False``
  - "bm25": This scheme includes a local tf component that increases asymptotically,
    so higher tfs have diminishing effects on the overall weight; a global idf
    component that can go *negative* for terms that appear in a sufficiently
    high proportion of docs; as well as a row-wise normalization that accounts for
    document length, such that terms in shorter docs hit the tf asymptote sooner
    than those in longer docs.
    Params: ``tf_type="bm25", apply_idf=True, idf_type="bm25", apply_dl=True``
  - "binary": This weighting scheme simply replaces all non-zero tfs with 1,
    indicating the presence or absence of a term in a particular doc. That's it.
    Params: ``tf_type="binary", apply_idf=False, apply_dl=False``
  
  Slightly altered versions of these "standard" weighting schemes are common,
  and may have better behavior in general use cases:
  
  - "lucene-style tfidf": Adds a doc-length normalization to the usual local
    and global components.
    Params: ``tf_type="linear", apply_idf=True, idf_type="smooth", apply_dl=True, dl_type="sqrt"``
  - "lucene-style bm25": Uses a smoothed idf instead of the classic bm25 variant
    to prevent weights on terms from going negative.
    Params: ``tf_type="bm25", apply_idf=True, idf_type="smooth", apply_dl=True, dl_type="linear"``
  
  Args:
      tf_type: Type of term frequency (tf) to use for weights' local component:
  
          - "linear": tf (tfs are already linear, so left as-is)
          - "sqrt": tf => sqrt(tf)
          - "log": tf => log(tf) + 1
          - "binary": tf => 1
  
      idf_type: Type of inverse document frequency (idf) to use for weights'
          global component:
  
          - "standard": idf = log(n_docs / df) + 1.0
          - "smooth": idf = log(n_docs + 1 / df + 1) + 1.0, i.e. 1 is added
            to all document frequencies, as if a single document containing
            every unique term was added to the corpus.
          - "bm25": idf = log((n_docs - df + 0.5) / (df + 0.5)), which is
            a form commonly used in information retrieval that allows for
            very common terms to receive negative weights.
          - None: no global weighting is applied to local term weights.
  
      dl_type: Type of document-length scaling to use for weights'
          normalization component:
  
          - "linear": dl (dls are already linear, so left as-is)
          - "sqrt": dl => sqrt(dl)
          - "log": dl => log(dl)
          - None: no normalization is applied to local (+global?) weights
  
      norm: If "l1" or "l2", normalize weights by the L1 or L2 norms, respectively,
          of row-wise vectors; otherwise, don't.
      min_df: Minimum number of documents in which a term must appear for it to be
          included in the vocabulary and as a column in a transformed doc-term matrix.
          If float, value is the fractional proportion of the total number of docs,
          which must be in [0.0, 1.0]; if int, value is the absolute number.
      max_df: Maximum number of documents in which a term may appear for it to be
          included in the vocabulary and as a column in a transformed doc-term matrix.
          If float, value is the fractional proportion of the total number of docs,
          which must be in [0.0, 1.0]; if int, value is the absolute number.
      max_n_terms: If specified, only include terms whose document frequency is within
          the top ``max_n_terms``.
      vocabulary_terms: Mapping of unique term string to unique term id, or
          an iterable of term strings that gets converted into such a mapping.
          Note that, if specified, vectorized outputs will include *only* these terms.
  
  Attributes:
      vocabulary_terms (dict[str, int]): Mapping of unique term string to unique
          term id, either provided on instantiation or generated by calling
          :meth:`Vectorizer.fit()` on a collection of tokenized documents.
class_docstr_tok: ["Transform","one","or","more","tokenized","documents","into","a","sparse","document-term","matrix","of","shape","(","#","docs",",","#","unique","terms",")",",","with","flexible","weighting\/normalization","of","values",".","Stream","a","corpus","with","metadata","from","disk",":","..","code-block",":",":","pycon",">",">",">","ds","=","textacy.datasets.CapitolWords","(",")",">",">",">","records","=","ds.records","(","limit=1000",")",">",">",">","corpus","=","textacy.Corpus","(","``","en_core_web_sm","''",",","data=records",")",">",">",">","print","(","corpus",")","Corpus","(","1000","docs",",","538397","tokens",")","Tokenize","and","vectorize","the","first","600","documents","of","this","corpus",":","..","code-block",":",":","pycon",">",">",">","tokenized_docs","=","(","...","(","term.lemma_","for","term","in","textacy.extract.terms","(","doc",",","ngs=1",",","ents=True",")",")","...","for","doc","in","corpus","[",":600","]",")",">",">",">","vectorizer","=","Vectorizer","(","...","tf_type=","''","linear","''",",","idf_type=","''","smooth","''",",","norm=","''","l2","''",",","...","min_df=3",",","max_df=0.95",")",">",">",">","doc_term_matrix","=","vectorizer.fit_transform","(","tokenized_docs",")",">",">",">","doc_term_matrix","<","600x4412","sparse","matrix","of","type","'","<","class","'numpy.float64","'",">","'","with","65210","stored","elements","in","Compressed","Sparse","Row","format",">","Tokenize","and","vectorize","the","remaining","400","documents","of","the","corpus",",","using","only","the","groups",",","terms",",","and","weights","learned","in","the","previous","step",":","..","code-block",":",":","pycon",">",">",">","tokenized_docs","=","(","...","(","term.lemma_","for","term","in","textacy.extract.terms","(","doc",",","ngs=1",",","ents=True",")",")","...","for","doc","in","corpus","[","600",":","]",")",">",">",">","doc_term_matrix","=","vectorizer.transform","(","tokenized_docs",")",">",">",">","doc_term_matrix","<","400x4412","sparse","matrix","of","type","'","<","class","'numpy.float64","'",">","'","with","36212","stored","elements","in","Compressed","Sparse","Row","format",">","Inspect","the","terms","associated","with","columns",";","they","'re","sorted","alphabetically",":","..","code-block",":",":","pycon",">",">",">","vectorizer.terms_list","[",":5","]","[","``",",","'","$","'",",","'","$","1","million","'",",","'","$","1.2","billion","'",",","'","$","10","billion","'","]","(","Btw",":","That","empty","string","should","n't","be","there",".","Somehow",",","spaCy","is","labeling","it","as","a","named","entity","...","?",")","If","known","in","advance",",","limit","the","terms","included","in","vectorized","outputs","to","a","particular","set","of","values",":","..","code-block",":",":","pycon",">",">",">","tokenized_docs","=","(","...","(","term.lemma_","for","term","in","textacy.extract.terms","(","doc",",","ngs=1",",","ents=True",")",")","...","for","doc","in","corpus","[",":600","]",")",">",">",">","vectorizer","=","Vectorizer","(","...","idf_type=","''","smooth","''",",","norm=","''","l2","''",",","...","min_df=3",",","max_df=0.95",",","...","vocabulary_terms=","[","``","president","''",",","``","bill","''",",","``","unanimous","''",",","``","distinguished","''",",","``","american","''","]",")",">",">",">","doc_term_matrix","=","vectorizer.fit_transform","(","tokenized_docs",")",">",">",">","doc_term_matrix","<","600x5","sparse","matrix","of","type","'","<","class","'numpy.float64","'",">","'","with","516","stored","elements","in","Compressed","Sparse","Row","format",">",">",">",">","vectorizer.terms_list","[","'american","'",",","'bill","'",",","'distinguished","'",",","'president","'",",","'unanimous","'","]","Specify","different","weighting","schemes","to","determine","values","in","the","matrix",",","adding","or","customizing","individual","components",",","as","desired",":","..","code-block",":",":","pycon",">",">",">","tokenized_docs","=","[","[","term.lemma_","for","term","in","textacy.extract.terms","(","doc",",","ngs=1",",","ents=True",")","]","for","doc","in","corpus","[",":600","]","]",">",">",">","doc_term_matrix","=","Vectorizer","(","...","tf_type=","''","linear","''",",","norm=None",",","min_df=3",",","max_df=0.95","...",")",".fit_transform","(","tokenized_docs",")",">",">",">","print","(","doc_term_matrix","[",":8",",","vectorizer.vocabulary_terms","[","``","$","''","]","]",".toarray","(",")",")","[","[","0","]","[","0","]","[","1","]","[","4","]","[","0","]","[","0","]","[","2","]","[","4","]","]",">",">",">","doc_term_matrix","=","Vectorizer","(","...","tf_type=","''","sqrt","''",",","dl_type=","''","sqrt","''",",","norm=None",",","min_df=3",",","max_df=0.95","...",")",".fit_transform","(","tokenized_docs",")",">",">",">","print","(","doc_term_matrix","[",":8",",","vectorizer.vocabulary_terms","[","``","$","''","]","]",".toarray","(",")",")","[","[","0",".","]","[","0",".","]","[","0.10660036","]","[","0.2773501","]","[","0",".","]","[","0",".","]","[","0.11704115","]","[","0.24806947","]","]",">",">",">","doc_term_matrix","=","Vectorizer","(","...","tf_type=","''","bm25","''",",","idf_type=","''","smooth","''",",","norm=None",",","min_df=3",",","max_df=0.95","...",")",".fit_transform","(","tokenized_docs",")",">",">",">","print","(","doc_term_matrix","[",":8",",","vectorizer.vocabulary_terms","[","``","$","''","]","]",".toarray","(",")",")","[","[","0",".","]","[","0",".","]","[","2.68009606","]","[","4.97732126","]","[","0",".","]","[","0",".","]","[","3.87124987","]","[","4.97732126","]","]","If","you","'re","not","sure","what","'s","going","on","mathematically",",",":","attr",":","`","Vectorizer.weighting","`","gives","the","formula","being","used","to","calculate","weights",",","based","on","the","parameters","set","when","initializing","the","vectorizer",":","..","code-block",":",":","pycon",">",">",">","vectorizer.weighting","'","(","tf","*","(","k","+","1",")",")","\/","(","k","+","tf",")","*","log","(","(","n_docs","+","1",")","\/","(","df","+","1",")",")","+","1'","In","general",",","weights","may","consist","of","a","local","component","(","term","frequency",")",",","a","global","component","(","inverse","document","frequency",")",",","and","a","normalization","component","(","document","length",")",".","Individual","components","may","be","modified",":","they","may","have","different","scaling","(","e.g",".","tf","vs.","sqrt","(","tf",")",")","or","different","behaviors","(","e.g",".","``","standard","''","idf","vs","bm25","'s","version",")",".","There","are","*","many","*","possible","weightings",",","and","some","may","be","better","for","particular","use","cases","than","others",".","When","in","doubt",",","though",",","just","go","with","something","standard",".","-","``","tf","''",":","Weights","are","simply","the","absolute","per-document","term","frequencies","(","tfs",")",",","i.e",".","value","(","i",",","j",")","in","an","output","doc-term","matrix","corresponds","to","the","number","of","occurrences","of","term","j","in","doc","i",".","Terms","appearing","many","times","in","a","given","doc","receive","higher","weights","than","less","common","terms",".","Params",":","``","tf_type=","''","linear","''",",","apply_idf=False",",","apply_dl=False","``","-","``","tfidf","''",":","Doc-specific",",","*","local","*","tfs","are","multiplied","by","their","corpus-wide",",","*","global","*","inverse","document","frequencies","(","idfs",")",".","Terms","appearing","in","many","docs","have","higher","document","frequencies","(","dfs",")",",","correspondingly","smaller","idfs",",","and","in","turn",",","lower","weights",".","Params",":","``","tf_type=","''","linear","''",",","apply_idf=True",",","idf_type=","''","smooth","''",",","apply_dl=False","``","-","``","bm25","''",":","This","scheme","includes","a","local","tf","component","that","increases","asymptotically",",","so","higher","tfs","have","diminishing","effects","on","the","overall","weight",";","a","global","idf","component","that","can","go","*","negative","*","for","terms","that","appear","in","a","sufficiently","high","proportion","of","docs",";","as","well","as","a","row-wise","normalization","that","accounts","for","document","length",",","such","that","terms","in","shorter","docs","hit","the","tf","asymptote","sooner","than","those","in","longer","docs",".","Params",":","``","tf_type=","''","bm25","''",",","apply_idf=True",",","idf_type=","''","bm25","''",",","apply_dl=True","``","-","``","binary","''",":","This","weighting","scheme","simply","replaces","all","non-zero","tfs","with","1",",","indicating","the","presence","or","absence","of","a","term","in","a","particular","doc",".","That","'s","it",".","Params",":","``","tf_type=","''","binary","''",",","apply_idf=False",",","apply_dl=False","``","Slightly","altered","versions","of","these","``","standard","''","weighting","schemes","are","common",",","and","may","have","better","behavior","in","general","use","cases",":","-","``","lucene-style","tfidf","''",":","Adds","a","doc-length","normalization","to","the","usual","local","and","global","components",".","Params",":","``","tf_type=","''","linear","''",",","apply_idf=True",",","idf_type=","''","smooth","''",",","apply_dl=True",",","dl_type=","''","sqrt","''","``","-","``","lucene-style","bm25","''",":","Uses","a","smoothed","idf","instead","of","the","classic","bm25","variant","to","prevent","weights","on","terms","from","going","negative",".","Params",":","``","tf_type=","''","bm25","''",",","apply_idf=True",",","idf_type=","''","smooth","''",",","apply_dl=True",",","dl_type=","''","linear","''","``","Args",":","tf_type",":","Type","of","term","frequency","(","tf",")","to","use","for","weights","'","local","component",":","-","``","linear","''",":","tf","(","tfs","are","already","linear",",","so","left","as-is",")","-","``","sqrt","''",":","tf","=",">","sqrt","(","tf",")","-","``","log","''",":","tf","=",">","log","(","tf",")","+","1","-","``","binary","''",":","tf","=",">","1","idf_type",":","Type","of","inverse","document","frequency","(","idf",")","to","use","for","weights'","global","component",":","-","``","standard","''",":","idf","=","log","(","n_docs","\/","df",")","+","1.0","-","``","smooth","''",":","idf","=","log","(","n_docs","+","1","\/","df","+","1",")","+","1.0",",","i.e",".","1","is","added","to","all","document","frequencies",",","as","if","a","single","document","containing","every","unique","term","was","added","to","the","corpus",".","-","``","bm25","''",":","idf","=","log","(","(","n_docs","-","df","+","0.5",")","\/","(","df","+","0.5",")",")",",","which","is","a","form","commonly","used","in","information","retrieval","that","allows","for","very","common","terms","to","receive","negative","weights",".","-","None",":","no","global","weighting","is","applied","to","local","term","weights",".","dl_type",":","Type","of","document-length","scaling","to","use","for","weights'","normalization","component",":","-","``","linear","''",":","dl","(","dls","are","already","linear",",","so","left","as-is",")","-","``","sqrt","''",":","dl","=",">","sqrt","(","dl",")","-","``","log","''",":","dl","=",">","log","(","dl",")","-","None",":","no","normalization","is","applied","to","local","(","+global","?",")","weights","norm",":","If","``","l1","''","or","``","l2","''",",","normalize","weights","by","the","L1","or","L2","norms",",","respectively",",","of","row-wise","vectors",";","otherwise",",","do","n't",".","min_df",":","Minimum","number","of","documents","in","which","a","term","must","appear","for","it","to","be","included","in","the","vocabulary","and","as","a","column","in","a","transformed","doc-term","matrix",".","If","float",",","value","is","the","fractional","proportion","of","the","total","number","of","docs",",","which","must","be","in","[","0.0",",","1.0","]",";","if","int",",","value","is","the","absolute","number",".","max_df",":","Maximum","number","of","documents","in","which","a","term","may","appear","for","it","to","be","included","in","the","vocabulary","and","as","a","column","in","a","transformed","doc-term","matrix",".","If","float",",","value","is","the","fractional","proportion","of","the","total","number","of","docs",",","which","must","be","in","[","0.0",",","1.0","]",";","if","int",",","value","is","the","absolute","number",".","max_n_terms",":","If","specified",",","only","include","terms","whose","document","frequency","is","within","the","top","``","max_n_terms","``",".","vocabulary_terms",":","Mapping","of","unique","term","string","to","unique","term","id",",","or","an","iterable","of","term","strings","that","gets","converted","into","such","a","mapping",".","Note","that",",","if","specified",",","vectorized","outputs","will","include","*","only","*","these","terms",".","Attributes",":","vocabulary_terms","(","dict","[","str",",","int","]",")",":","Mapping","of","unique","term","string","to","unique","term","id",",","either","provided","on","instantiation","or","generated","by","calling",":","meth",":","`","Vectorizer.fit","(",")","`","on","a","collection","of","tokenized","documents","."]
docstr: Sort terms in ``vocabulary`` alphabetically, modifying the vocabulary
  in-place, and returning a correspondingly reordered ``matrix`` along
  its rows or columns, depending on ``axis``.
  
  Args:
      matrix
      vocabulary
      axis
docstr_tok: ["Sort","terms","in","``","vocabulary","``","alphabetically",",","modifying","the","vocabulary","in-place",",","and","returning","a","correspondingly","reordered","``","matrix","``","along","its","rows","or","columns",",","depending","on","``","axis","``",".","Args",":","matrix","vocabulary","axis"]
code: def _sort_vocab_and_matrix(
      self,
      matrix: sp.csr_matrix,
      vocabulary: dict[str, int],
      axis: Literal["rows", 0] | Literal["columns", 1],
  ) -> sp.csr_matrix:
      """
      Sort terms in ``vocabulary`` alphabetically, modifying the vocabulary
      in-place, and returning a correspondingly reordered ``matrix`` along
      its rows or columns, depending on ``axis``.
  
      Args:
          matrix
          vocabulary
          axis
      """
      sorted_vocab = sorted(vocabulary.items())
      new_idx_array = np.empty(len(sorted_vocab), dtype=np.int32)
      for new_idx, (term, old_idx) in enumerate(sorted_vocab):
          new_idx_array[new_idx] = old_idx
          vocabulary[term] = new_idx
      # use fancy indexing to reorder rows or columns
      if axis == "rows" or axis == 0:
          return matrix[new_idx_array, :]
      elif axis == "columns" or axis == 1:
          return matrix[:, new_idx_array]
      else:
          raise ValueError(
              errors.value_invalid_msg(
                  "axis", axis, {"rows", "columns", 0, 1}
              )
          )
  
imports: from __future__ import annotations
  import collections
  import collections.abc
  import operator
  from array import array
  from typing import DefaultDict, Iterable, Literal, Optional, Union
  import numpy
  import scipy.sparse
  from sklearn.preprocessing import normalize
  from ..None import errors
  from .matrix_utils import filter_terms_by_df, get_doc_lengths, get_inverse_doc_freqs
label_desc: Use image node_id 12 for calling the Vectorizer obj's underlying member method code with example usage: obj._sort_vocab_and_matrix(matrix, vocabulary, axis) and returns: matrix, matrix