5 · Bring your own corpus¶

Every tutorial so far has started from a benchmark. Your data probably lives in a CSV file or a DataFrame, and it may not be in English. This tutorial covers the checklist for using your own corpus:

1. Format: from a DataFrame to .txt + .meta with save_corpus().
2. Language: stop words and non-ASCII scripts.
3. Encoding: files that are not UTF-8.
4. Size: will the graph fit in memory?

We use a small Portuguese corpus written inline. Runs on CPU in a few seconds.

#	Tutorial	Question it answers
1	Text as a graph	What does the graph look like, and why does it help?
2	Your first classifier	How do I go from raw text to test accuracy?
3	Choosing a graph builder	How should words be connected?
4	Refining the graph	Can a smaller graph do as well?
5	Bring your own corpus	How do I use my own (non-English) data?

In [ ]:

%pip install -q text4gcn

%pip install -q text4gcn

In [2]:

import shutil
from pathlib import Path

import pandas as pd
from sklearn.model_selection import train_test_split

from text4gcn import GNN, TextPipeline
from text4gcn.builder import Frequency
from text4gcn.datasets import save_corpus

import shutil from pathlib import Path import pandas as pd from sklearn.model_selection import train_test_split from text4gcn import GNN, TextPipeline from text4gcn.builder import Frequency from text4gcn.datasets import save_corpus

1. From a DataFrame to Text4GCN files¶

A typical labelled dataset has a text column and a label column, and either a split column or no split at all. If there is no split, create one with scikit-learn and stratify it, so that every class appears in both train and test.

In [3]:

df = pd.DataFrame({
    "texto": [
        "O atacante marcou dois gols e o time venceu o clássico no estádio lotado",
        "O técnico escalou a equipe titular para a final do campeonato",
        "A torcida comemorou o título depois da vitória nos pênaltis",
        "O goleiro defendeu o pênalti e garantiu a classificação do time",
        "O campeonato brasileiro terá a última rodada no domingo",
        "A seleção treinou finalização antes do jogo das eliminatórias",
        "O zagueiro fez um gol de cabeça no segundo tempo da partida",
        "A equipe venceu fora de casa e assumiu a liderança do campeonato",
        "Misture a farinha com os ovos e o açúcar até formar uma massa lisa",
        "Asse o bolo no forno pré-aquecido por quarenta minutos",
        "Refogue a cebola e o alho no azeite antes de colocar o arroz",
        "Tempere o frango com limão, sal e ervas frescas",
        "Bata as claras em neve e incorpore delicadamente à massa do bolo",
        "Cozinhe o feijão na panela de pressão com louro e alho",
        "Sirva o peixe grelhado com arroz, salada e azeite",
        "Leve a massa ao forno e asse até dourar",
    ],
    "categoria": ["esporte"] * 8 + ["culinária"] * 8,
})

_, test_index = train_test_split(df.index, test_size=0.25, stratify=df.categoria, random_state=0)
df["split"] = ["test" if i in test_index else "train" for i in df.index]
df.groupby(["categoria", "split"]).size().unstack()

df = pd.DataFrame({ "texto": [ "O atacante marcou dois gols e o time venceu o clássico no estádio lotado", "O técnico escalou a equipe titular para a final do campeonato", "A torcida comemorou o título depois da vitória nos pênaltis", "O goleiro defendeu o pênalti e garantiu a classificação do time", "O campeonato brasileiro terá a última rodada no domingo", "A seleção treinou finalização antes do jogo das eliminatórias", "O zagueiro fez um gol de cabeça no segundo tempo da partida", "A equipe venceu fora de casa e assumiu a liderança do campeonato", "Misture a farinha com os ovos e o açúcar até formar uma massa lisa", "Asse o bolo no forno pré-aquecido por quarenta minutos", "Refogue a cebola e o alho no azeite antes de colocar o arroz", "Tempere o frango com limão, sal e ervas frescas", "Bata as claras em neve e incorpore delicadamente à massa do bolo", "Cozinhe o feijão na panela de pressão com louro e alho", "Sirva o peixe grelhado com arroz, salada e azeite", "Leve a massa ao forno e asse até dourar", ], "categoria": ["esporte"] * 8 + ["culinária"] * 8, }) _, test_index = train_test_split(df.index, test_size=0.25, stratify=df.categoria, random_state=0) df["split"] = ["test" if i in test_index else "train" for i in df.index] df.groupby(["categoria", "split"]).size().unstack()

Out[3]:

split	test	train
categoria
culinária	2	6
esporte	2	6

save_corpus validates the input before writing anything. It checks that the lists have equal lengths, that every split is train or test, that the labels contain no tabs or newlines, and that both splits are present. It collapses line breaks inside each text and writes UTF-8.

In [4]:

NAME, DATA_DIR = "noticias_receitas", Path("data")
shutil.rmtree(DATA_DIR, ignore_errors=True)

save_corpus(DATA_DIR, NAME, texts=df.texto, labels=df.categoria, splits=df.split)

NAME, DATA_DIR = "noticias_receitas", Path("data") shutil.rmtree(DATA_DIR, ignore_errors=True) save_corpus(DATA_DIR, NAME, texts=df.texto, labels=df.categoria, splits=df.split)

Out[4]:

(PosixPath('data/noticias_receitas.txt'),
 PosixPath('data/noticias_receitas.meta'))

2. Language: stop words and non-ASCII text¶

Two TextPipeline options matter for languages other than English:

• language: the NLTK stop-word list to remove, for example "portuguese", "spanish" or "arabic".
• ascii_only: the original TextGCN cleaning keeps only A-Za-z0-9. That breaks accented words (clássico becomes cl ssico) and erases scripts such as Arabic completely. Set ascii_only=False to keep letters of any script.

Compare the vocabulary produced by each setting:

In [5]:

def vocabulary(**options) -> set[str]:
    for folder in DATA_DIR.glob(f"{NAME}.*"):
        if folder.is_dir():
            shutil.rmtree(folder)                 # start each run from the input files only
    TextPipeline(dataset_name=NAME, dataset_path=DATA_DIR, rare_count=1, **options).execute()
    vocab_path = DATA_DIR / f"{NAME}.shuffled" / f"{NAME}.vocab"
    return set(vocab_path.read_text(encoding="utf-8").split())

ascii_vocab = vocabulary(language="portuguese")
unicode_vocab = vocabulary(language="portuguese", ascii_only=False)

print("only with ascii_only=True :", sorted(ascii_vocab - unicode_vocab))
print("only with ascii_only=False:", sorted(unicode_vocab - ascii_vocab))

def vocabulary(**options) -> set[str]: for folder in DATA_DIR.glob(f"{NAME}.*"): if folder.is_dir(): shutil.rmtree(folder) # start each run from the input files only TextPipeline(dataset_name=NAME, dataset_path=DATA_DIR, rare_count=1, **options).execute() vocab_path = DATA_DIR / f"{NAME}.shuffled" / f"{NAME}.vocab" return set(vocab_path.read_text(encoding="utf-8").split()) ascii_vocab = vocabulary(language="portuguese") unicode_vocab = vocabulary(language="portuguese", ascii_only=False) print("only with ascii_only=True :", sorted(ascii_vocab - unicode_vocab)) print("only with ascii_only=False:", sorted(unicode_vocab - ascii_vocab))

text4gcn · TextPipeline("noticias_receitas") · language portuguese · validation 10%
  ✓ clean     16 documents · 96 words kept (rare_count=1)  (<0.1 s)
  ✓ shuffle   train 12 · test 4 documents (seed 1234)  (<0.1 s)
  ✓ words     96 words · 19 with a WordNet definition  (1.3 s)
  ✓ features  train 11 · validation 1 · test 4 documents · 2 classes  (<0.1 s)
text4gcn · TextPipeline("noticias_receitas") · language portuguese · validation 10%
  ✓ clean     16 documents · 88 words kept (rare_count=1)  (<0.1 s)
  ✓ shuffle   train 12 · test 4 documents (seed 1234)  (<0.1 s)
  ✓ words     88 words · 11 with a WordNet definition  (<0.1 s)
  ✓ features  train 11 · validation 1 · test 4 documents · 2 classes  (<0.1 s)
only with ascii_only=True : ['at', 'cabe', 'car', 'cl', 'classifica', 'cnico', 'dio', 'eliminat', 'est', 'feij', 'finaliza', 'lideran', 'lim', 'ltima', 'nalti', 'naltis', 'p', 'pr', 'press', 'ria', 'rias', 'sele', 'ssico', 't', 'ter', 'tulo', 'vit']
only with ascii_only=False: ['açúcar', 'cabeça', 'classificação', 'clássico', 'eliminatórias', 'estádio', 'feijão', 'finalização', 'liderança', 'limão', 'pressão', 'pré', 'pênalti', 'pênaltis', 'seleção', 'técnico', 'título', 'vitória', 'última']

With ascii_only=True, words lose their accented letters and break into meaningless fragments. With ascii_only=False, clássico, pênalti and seleção survive as words. The last run used ascii_only=False, so the folders on disk now hold the right version.

3. Encoding¶

TextPipeline reads the corpus as UTF-8 by default. Older datasets are often Latin-1 or Windows-1252. In that case reading fails with a message telling you to pass the encoding, for example TextPipeline(..., encoding="latin1"). encoding applies to the .txt file; the .meta file must always be UTF-8. save_corpus always writes UTF-8, so data that went through it is safe.

4. Will the graph fit?¶

The graph has one node per document plus one per vocabulary word. During training, the normalised adjacency and the identity node features are both kept as dense $N \times N$ matrices, $N$ being the number of nodes. Peak memory therefore grows with the square of the node count. We measured about 20 bytes per pair of nodes (3.9 GiB for the 14.5k-node graph of tutorial 2):

In [6]:

def peak_memory_gib(n_documents: int, vocabulary_size: int) -> float:
    """Rule of thumb for peak training memory: ~20 bytes per pair of nodes."""
    return (n_documents + vocabulary_size) ** 2 * 20 / 2**30

for docs, words in [(len(df), len(unicode_vocab)), (3_600, 11_000), (7_700, 7_700), (18_800, 42_000)]:
    print(f"{docs:>6,} documents + {words:>6,} words -> ~{peak_memory_gib(docs, words):5.1f} GiB")

def peak_memory_gib(n_documents: int, vocabulary_size: int) -> float: """Rule of thumb for peak training memory: ~20 bytes per pair of nodes.""" return (n_documents + vocabulary_size) ** 2 * 20 / 2**30 for docs, words in [(len(df), len(unicode_vocab)), (3_600, 11_000), (7_700, 7_700), (18_800, 42_000)]: print(f"{docs:>6,} documents + {words:>6,} words -> ~{peak_memory_gib(docs, words):5.1f} GiB")

    16 documents +     88 words -> ~  0.0 GiB
 3,600 documents + 11,000 words -> ~  4.0 GiB
 7,700 documents +  7,700 words -> ~  4.4 GiB
18,800 documents + 42,000 words -> ~ 68.9 GiB

Rows 2–4 correspond roughly to the 20NG subset of the previous tutorials, R8, and the full 20NG. If the estimate is too big, raise rare_count (fewer words), use a subset of the documents, or prune the graph (tutorials 3 and 4).

Train¶

From here on, the steps are the same as for any benchmark:

In [7]:

Frequency(dataset_name=NAME, dataset_path=DATA_DIR).tfidf()
result = GNN(dataset_name=NAME, dataset_path=DATA_DIR, log_dir=Path("logs"), epochs=100, device="cpu").fit()
print(f"test accuracy: {result.test_accuracy:.2%} on {(df.split == 'test').sum()} documents")

Frequency(dataset_name=NAME, dataset_path=DATA_DIR).tfidf() result = GNN(dataset_name=NAME, dataset_path=DATA_DIR, log_dir=Path("logs"), epochs=100, device="cpu").fit() print(f"test accuracy: {result.test_accuracy:.2%} on {(df.split == 'test').sum()} documents")

text4gcn · Frequency.tfidf("noticias_receitas") · window 20
  ✓ graph     104 nodes · 673 edges (570 word–word) · density 6.22% · 8.7 KiB  (<0.1 s)
text4gcn · GNN("noticias_receitas") · frequency.tfidf · cpu · seed 1234
  graph     104 nodes: 11 train · 1 validation · 4 test documents + 88 words
  model     GCN 104 → 200 → 2 · 21,200 parameters · dropout 0.5 · lr 0.02
  ✓ train     100 epochs (maximum reached) · best epoch 100 (val_loss 0.6865, val_acc 100.0%)  (0.6 s)
  ✓ test      accuracy 75.0% · macro-F1 73.3% · loss 0.8141 · 4 documents  (<0.1 s)

        class  precision  recall    f1  support
      esporte      1.000   0.500 0.667        2
    culinária      0.667   1.000 0.800        2

  model     logs/20261016-234039522.pth
test accuracy: 75.00% on 4 documents

Sixteen sentences are only enough to check that the pipeline works, not to measure a model. With a real corpus, the first classifier and graph builder tutorials apply unchanged.

Checklist¶

Format	save_corpus(path, name, texts, labels, splits), with splits set to train/test and stratified
Language	TextPipeline(language="portuguese", ascii_only=False)
Encoding	UTF-8 by default; encoding="latin1" for legacy files
Re-runs	delete the <name>.*/ output folders first, so that no output of an older run is reused
Size	about $(\text{documents} + \text{words})^2 \times 20$ bytes at peak: keep it below your GPU/RAM
Splits	test documents are part of the graph (transductive); new documents mean retraining