add a note about history

1 files changed, 9 insertions, 0 deletions

diff --git a/README.md b/README.md
index c6f789f..e4b9398 100644
--- a/README.md
+++ b/README.md
@@ -104,6 +104,8 @@ below (note the invisible space on the lines with dots):
 .\n
 ```
 
+### Custom tokenizers
+
 Users may supply any tokenizer via option `-F`. The script below produces
 line-size tokens for demonstration (in turn, `werge` will do the usual line
 merges), and can be used e.g. via `-F ./tokenize.py`:
@@ -119,6 +121,13 @@ for l in sys.stdin.readlines():
         print('/'+l.replace('\\','\\\\'))
 ```
 
+### History
+
+I previously made an attempt to solve this in `adiff` software, which failed
+because the approach was too complex. Before that, the issue was tackled by
+Arek Antoniewicz on MFF CUNI, who used regex-edged DFAs (REDFAs) to construct
+user-specifiable tokenizers in a pretty cool way.
+
 ## Installation
 
 ```sh