From f5f206765cf05f59b6482bca04a9913aef5330d5 Mon Sep 17 00:00:00 2001
From: Mirek Kratochvil <exa.exa@gmail.com>
Date: Fri, 18 Jul 2025 15:58:24 +0200
Subject: [PATCH] add a note about history

---
 README.md | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/README.md b/README.md
index c6f789f..e4b9398 100644
--- a/README.md
+++ b/README.md
@@ -104,6 +104,8 @@ below (note the invisible space on the lines with dots):
 .\n
 ```
 
+### Custom tokenizers
+
 Users may supply any tokenizer via option `-F`. The script below produces
 line-size tokens for demonstration (in turn, `werge` will do the usual line
 merges), and can be used e.g. via `-F ./tokenize.py`:
@@ -119,6 +121,13 @@ for l in sys.stdin.readlines():
         print('/'+l.replace('\\','\\\\'))
 ```
 
+### History
+
+I previously made an attempt to solve this in `adiff` software, which failed
+because the approach was too complex. Before that, the issue was tackled by
+Arek Antoniewicz on MFF CUNI, who used regex-edged DFAs (REDFAs) to construct
+user-specifiable tokenizers in a pretty cool way.
+
 ## Installation
 
 ```sh