README.md

---
license: mit
tags:
- misspelling
- generator
- python
- ipynb
---

# Misspelling Generator 

A misspelling words generator of the 466k words from data [provider](https://github.com/dwyl/english-words) written `words.txt`. For demonstation, we only use 7 letters combination minimum to generate:
```
  Words processed  : 125,414
  Lines written    : 173,110,626
  Output file      : misspellings_permutations.txt
  File size        : 2.53 GB
```
depending your storage, you could do more litter combination limit, just configure the `MAX_WORD_LEN` in the script.

- use `generate_permutations_colab.py` or `google_collab_173MSW.ipynb` to generate 173,110,626 misspelling words from 125,414 processed datasets; the output is downloadable at [HuggingFace](https://huggingface.co/datasets/algorembrant/generated-misspelling-words/blob/main/173Million-misspelling-words.txt)
- use `generate_typos_colab.py` or `google_collab_263MSW.ipynb` to  generate 26,636,990 misspelling words from 415,701 processed datasets; the output is downloadable at [Hugging Face](https://huggingface.co/datasets/algorembrant/generated-misspelling-words/blob/main/26Million-misspelling-words.txt)
  
<br><br><br>
### Option 1 — use ``generate_typos_local.py` (Run Locally)

Generates **realistic typo variants** using 4 strategies:

| Strategy | Example (`hello`) | Variants |
|---|---|---|
| Adjacent swap | `hlelo`, `helol` | n−1 per word |
| Char deletion | `hllo`, `helo`, `hell` | n per word |
| Char duplication | `hhello`, `heello` | n per word |
| Keyboard proximity | `gello`, `jello`, `hwllo` | varies |

- Processes only **pure-alpha words** with length ≥ 3
- Produces roughly **10–50 typos per word** → ~5M–20M lines total
- Output: data/misspellings.txt in `misspelling=correction` format

**To run:**
```
python generate_typos_local.py
```

---

### Option 2 — use `generate_permutations_colab.py` (Google Colab)

Generates **ALL letter permutations** of each word. Key config at the top of the file:

```python
MAX_WORD_LEN = 7   # ← CRITICAL control knob
```

---

## Google Colab Education

### What Is Google Colab?

Google Colab gives you a **free Linux VM** with Python pre-installed. You get:

| Resource | Free Tier | Colab Pro ($12/mo) |
|---|---|---|
| **Disk** | ~78 GB (temporary) | ~225 GB (temporary) |
| **RAM** | ~12 GB | ~25-50 GB |
| **GPU** | T4 (limited) | A100/V100 |
| **Runtime limit** | ~12 hours, then VM resets | ~24 hours |
| **Google Drive** | 15 GB (persistent) | 15 GB (same) |

> [!IMPORTANT]
> Colab disk is **ephemeral** — when the runtime disconnects, all files on the VM are deleted. Only Google Drive persists.

### Step-by-Step: Running Option 2 on Colab

**Step 1 — Open Colab**
Go to [colab.research.google.com](https://colab.research.google.com) → **New Notebook**

**Step 2 — Upload [words.txt]**
```python
# Cell 1
from google.colab import files
uploaded = files.upload()   # select words.txt from your PC
```

**Step 3 — (Optional) Mount Google Drive for persistent storage**
```python
# Cell 2
from google.colab import drive
drive.mount('/content/drive')

# Then change OUTPUT_PATH in the script to:
# '/content/drive/MyDrive/misspellings_permutations.txt'
```

**Step 4 — Paste & run the script**
Copy the entire contents of `generate_permutations_colab.py` into a new cell. Adjust `MAX_WORD_LEN` as needed, then run.

**Step 5 — Download the result**
```python
# If saved to VM disk:
files.download('misspellings_permutations.txt')

# If saved to Google Drive: just access it from drive.google.com
```

### Scale Reference

> [!CAUTION]
> Full permutations grow at **n! (factorial)** rate. Here's what to expect:

| `MAX_WORD_LEN` | Max perms/word | Est. total output |
|---|---|---|
| 5 | 120 | ~200 MB |
| 6 | 720 | ~1–2 GB |
| **7** | **5,040** | **~5–15 GB** ← recommended start |
| 8 | 40,320 | ~50–150 GB |
| 9 | 362,880 | ~500 GB – 1 TB |
| 10 | 3,628,800 | ~5–50 TB ← impossible |

> [!TIP]
> **Start with `MAX_WORD_LEN = 6` or `7`**, check the output size, then decide if you want to go higher. The script has a built-in safety check that aborts if the estimated size exceeds 70 GB.

### Pro Tips for Colab

- **Keep the browser tab open** — Colab disconnects if idle too long
- **Use `Ctrl+Shift+I` → Console** and paste `setInterval(function(){document.querySelector("colab-connect-button").click()}, 60000)` to prevent idle disconnects
- **For very large outputs**, write directly to Google Drive so you don't lose data on disconnect
- **CPU-only is fine** for this script — permutation generation is CPU-bound, not GPU