Salvatore Prochilo
All projects
completed

Document Image Binarization using Fast Fourier Convolution

MSc thesis: a fully convolutional network that uses Fast Fourier Convolutions to separate ink from paper in degraded historical documents, evaluated on ten DIBCO benchmarks.

Three panels of a handwritten affidavit: the original with writing from the back of the sheet showing through, the clean ground truth, and the network output, almost identical to the ground truth.
Role
MSc thesis — implementation, training and evaluation
Organization
UNIMORE – AImageLab
Period
2023
  • computer vision
  • document analysis
  • binarization
  • semantic segmentation
  • Fast Fourier Convolution
  • PyTorch
repo

Context

My Master’s thesis in Computer Engineering, Document Image Binarization with Fast Fourier Convolution, at the University of Modena and Reggio Emilia, at AImageLab, supervised by Prof. Rita Cucchiara, with Prof. Silvia Cascianelli and Vittorio Pippi as co-supervisors.

Archives scan old letters, ledgers and books by the million. Before a computer can read them (text recognition, layout analysis, search), it has to decide, pixel by pixel, what is ink and what is paper. That step is binarization, and on damaged documents it is far from solved.

The problem

  • Every page is degraded differently. Faded ink, stains, ink from the back of the sheet showing through (bleed-through), shadows and uneven lighting, often on the same page.
  • Global and local at once. To tell a stain from a stroke, a model needs the context of the whole page. To keep thin strokes intact, it needs pixel-level precision. Classic thresholds (Otsu, Sauvola) use only local statistics; standard CNNs have a small, fixed receptive field; Vision Transformers see globally but are heavy.
  • Very little data. The standard benchmarks (DIBCO and H-DIBCO, 2009–2019) contain 10 to 20 annotated images each: 139 images in total.

Approach

Network diagram. A 256 × 256 colour patch goes through an encoder of three strided convolutions, nine residual blocks built on Fast Fourier Convolutions, and a decoder of three transposed convolutions, giving an ink probability map thresholded into a binary image. Inside each Fast Fourier Convolution, a quarter of the channels go through an ordinary 3 × 3 convolution and three quarters go through a spectral path (FFT, 1 × 1 convolution, inverse FFT) that sees the whole patch at once.

Fast Fourier Convolutions for global context. I treated binarization as segmentation and built on LaMa, an inpainting network based on Fast Fourier Convolutions (FFC). In an FFC block, part of the channels are transformed with an FFT, mixed, and transformed back. A convolution in the frequency domain touches every pixel of the patch, so the network sees the whole patch from the first block, far more cheaply than self-attention. The idea fits documents well: paper texture and lines of text are close to periodic patterns. To my knowledge, FFCs had not been used for document binarization before.

I adapted the network to the task: RGB input, a single-channel ink map as output, nine FFC residual blocks with 75% of the channels on the spectral path, trained with binary cross-entropy (better than MSE in my tests: 19.78 against 19.36 dB PSNR on H-DIBCO 2018).

Patch merging: inventing new degradations. With so few images, the model would memorise the degradations it has seen. On top of the usual augmentations (rotation, flips, colour jitter), I combined two random training patches by taking their pixel-wise minimum, and did the same with their ground truths. Ink and stains are dark, so both survive: the result is a realistic page with a mix of damage that does not exist in the dataset.

Patch merging. Top row: a faded handwritten patch, a stained handwritten patch, and their pixel-wise minimum, which keeps the ink of both and the stain of the second. Bottom row: the corresponding binary ground truths, merged the same way.

Two training patches (left, centre) and their merge (right), with the matching ground truths below. Patches from DIBCO 2013 and DIBCO 2011.

A strict protocol. I used every DIBCO and H-DIBCO edition from 2009 to 2019 (plus the NoisyOffice and Palm Leaf datasets for training). For each year, the model was trained on all the other editions and tested on the held-out one, and scored with the standard metrics: F-measure, PSNR and DRD (a measure of visual distortion).

Results

Three panels of a handwritten affidavit. Left: the original, a grey page where the writing from the back of the sheet shows through as a faint tangle of lines. Centre: the ground truth, with only the front-side writing in black on white. Right: the network output, almost identical to the ground truth, with the back-side writing removed.

Input · ground truth · output. Bleed-through: the writing on the back of the sheet is removed, the signature is kept. DIBCO 2009.

Three panels of a printed newspaper page in Slovenian. Left: the original, with a large brown stain over the right half. Centre: the ground truth, clean black text. Right: the network output: the text is clean, including the lines under the stain, but the last word of the title, under the darkest part of the stain, is partly lost, and some specks remain in the top-right corner.

A dark stain on printed text. The lines under the stain are recovered; the word under the darkest part is only partly, and a few specks remain at the edge. DIBCO 2013.

Best result on the hardest benchmark. On H-DIBCO 2018, where Otsu’s method reaches only 9.7 dB PSNR, the network with 512 × 512 patches scored better than all the methods compared in the thesis, including transformer-based ones:

H-DIBCO 2018 F-measure ↑ PSNR ↑ DRD ↓
Otsu 51.45 9.74 59.07
Contest winner 88.34 19.11 4.92
GDB (gated convolutions) 91.09 19.92 3.07
Text-DIAE (transformer) 91.32 19.95 3.21
This work, 256 × 256 patches 91.03 19.78 3.02
This work, 512 × 512 patches 92.13 20.38 2.51

Larger patches helped (+0.6 dB PSNR, DRD from 3.02 to 2.51): more context is exactly what the spectral path is meant to use.

Competitive, not state of the art everywhere. Across the eight held-out editions from 2009 to 2017, PSNR ranges from 18.4 to 21.5 dB and F-measure from 90.9 to 96.0, far above classic thresholds. On DIBCO 2011, H-DIBCO 2012 and H-DIBCO 2014, recent transformer and CNN methods remain ahead (for example 21.10 against 23.66 dB on H-DIBCO 2012).

Read with care. As is common in DIBCO papers, the checkpoint was chosen on the held-out year, and the patch size and loss were compared on H-DIBCO 2018 itself. There was no separate validation split, so the numbers are slightly optimistic.

Where it fails

Three panels of a handwritten letter. Left: the original, with a water stain whose dark edge crosses two lines of text. Centre: the ground truth, with all words in black. Right: the network output: words outside the stain are clean, but inside the stain several words are broken or missing.

A water stain over faded ink. Outside the stain the output matches the ground truth; inside it, words such as “sum of money” are broken or lost. H-DIBCO 2012.

When ink and stain have almost the same colour, the network tends to call both background. The other weak point is at tile borders: pages are processed in non-overlapping patches, and errors appear where patches with different damage meet.

Afterwards

AImageLab later forked my codebase (February 2023), and the lab’s researchers continued this line of work, published as FourBi at ICDAR 2024.

Stack

Python, PyTorch, torchvision, PyTorch-Ignite (metrics), Weights & Biases (experiment tracking), OpenCV and scikit-image (classic baselines).

What I’d do next

  • Overlapping tiles at inference, blending predictions, to remove the seams at patch borders.
  • A separate validation split, so the held-out year is used only once, for the final score.
  • Post-processing of the binary mask (morphological operations), as proposed in the thesis, to clean isolated specks.

Images: from the DIBCO 2009 (Gatos, Ntirogiannis & Pratikakis, ICDAR 2009), DIBCO 2011 (Pratikakis, Gatos & Ntirogiannis, ICDAR 2011), H-DIBCO 2012 (Pratikakis, Gatos & Ntirogiannis, ICFHR 2012) and DIBCO 2013 (Pratikakis, Gatos & Ntirogiannis, ICDAR 2013) benchmark datasets. Outputs are from my model, as published in the thesis.