HerBERT reaches 0.986 against the baseline's 0.944

• 0.986
  Macro-F1, HerBERT
  95% interval 0.976–0.994, against 0.944 for the baseline
• 38 / 7
  Reviews it settles either way
  it is right where the baseline is wrong / wrong where the baseline is right, over 45 disagreements — p < 0.0001
• 0.980
  Cascade at 20% on the GPU
  85% of what the transformer adds, from 137 of 684 reviews
• 49 min
  What the transformer cost
  on a GeForce GTX 1050, against 3 s on a CPU

What the corpus score is, and how tightly it is pinned

Macro-F1 rather than accuracy, because the classes are imbalanced: answering negative to everything would score respectably on accuracy and badly here. The figure below it is the one that is usually left off — a score on 684 reviews is a sample, and the interval says how much of the third decimal is real.

Macro-F1 0.944, and 95% of 2000 bootstrap resamples of the test rows fall between 0.926 and 0.961. Any comparison narrower than 0.035 is a comparison this test set cannot make.

What that is worth depends on what it is read against, so the floors are on the page rather than left to the reader's intuition: always the majority class scores 0.221 macro-F1 at 0.496 accuracy, random, matching the training prior scores 0.317 macro-F1 at 0.367 accuracy. The first of those is the whole argument for the headline metric: it is a model that has learned nothing, and accuracy still flatters it.

Is the transformer better?

The intervals are not the comparison. Both models answer the same 684 reviews, so the question is which reviews they disagree on: HerBERT is right on 38 the baseline misses, the baseline is right on 7 HerBERT misses, and the remaining 639 they answer alike. An exact McNemar test on those 45 disagreements gives p < 0.0001.

How much of this score is data rather than model

Before asking what a transformer would add, it is worth asking what the corpus adds. The same pipeline is refit from scratch on stratified subsamples of the training split and scored on the untouched test split every time.

1200 labelled reviews — 23% of the corpus — already land within 0.019 macro-F1 of what all 5264 produce. The remaining 4064 annotations buy the last 1.9 points. Whether that is worth paying for is a budget question, but it is the same question as "should we fine-tune a transformer", asked where the answer is cheaper to test.

What the cheap side of that trade costs, measured rather than asserted: the full pipeline fits in 3.2 s on Intel64 Family 6 Model 158 Stepping 10, GenuineIntel (CPU), the saved model is 3.2 MB, and it classifies 5921 reviews a second. The other side of the trade, measured on the same task rather than described: fine-tuning HerBERT took 49 minutes on a GeForce GTX 1050 — 927× the training time, over 4 epochs of the same 5264 reviews, and it wants that card to serve as well as to train. Whether the difference in score is worth the difference in bill is the question this page exists to put numbers on.

Where a bag of words slips

The corpus score says how the model does on reviews like the ones it was trained on. It says nothing about the phenomena a lexical model is supposed to be bad at. This section used to assert that negation and irony are its weak points and illustrate the claim with three unscored sentences; it now measures it.

80 sentences, written for this page in the four domains PolEmo draws from, plus 158 derived by re-spelling the same sentences — diacritics stripped, and one adjacent-character swap in the longest word. No PolEmo text appears here; the corpus is non-commercial and is not redistributed by this repository.

The cells are built so that no single answer can win them. Every ironic sentence has a genuine counterpart in the same enthusiastic register, down to minimal pairs — "bateria wytrzymała całe trzy godziny" against "całe trzy dni" — because a probe made only of ironic praise would be scored 20 of 20 by a model that answers negative to everything. The negation cell carries cues in both directions for the same reason.

6 of 96 misses, in the order they were scored rather than picked for effect.

Where the misses go

A count of correct answers cannot tell a model that misreads irony apart from one that answers negative to everything — both score the same. The direction of the errors can, which is why it is a column rather than a sentence.

The eight most frequent of 26 distinct should-be/answered pairs, over all 96 misses including the derived cells.

Is that about length, or about our sentences?

The section above is the page's own claim, and it rests on sentences we wrote. The obvious objection is that the probe measures our writing rather than its length. PolEmo is where to put that question: nobody here chose its reviews, and the same predictions that produced the headline score can be cut by how long each one is. Whether the corpus is in a position to answer is the first thing the cut has to establish.

The corpus cannot settle it. PolEmo is made of written reviews — the median one in this split is 119 words — and only 3 of its 684 test reviews are as short as the probe's sentences (9 words at the median). The length the claim is about is a length this corpus barely contains, which is why the sentences had to be written rather than sampled. Across the lengths it does cover, counting only groups of at least 50 reviews, the score moves by -0.011 macro-F1 between the shortest and the longest of them — inside the interval around the score itself, so not a direction at all. So the honest reading of the section above is narrower than "short text breaks it": on sentence-length Polish reviews of the kind this corpus does not contain, the model is not the 0.94 model. What separates the two — length, register, or the distance between a review and a constructed sentence — this page cannot take apart, and does not claim to.

What the model keys on

A linear model has two advantages over a transformer, and only one of them is the compute bill. These are the 10 heaviest coefficients per class — the model's own reasons, not an explanation reconstructed after the fact. They are weights, not review text.

Worth reading against the cells above. Where a class's heaviest terms are evaluative, the model has learned sentiment; where they are topical — the vocabulary of the domains those reviews happen to come from — it has learned to recognise the subject matter, and will carry that habit onto any sentence about the same subject. A single token weighted for one class also cannot be conditioned on what follows it, which is what the negation cell measures.

Knowing when not to answer

A system that can run both models does not have to choose one. The cheap model answers what it is sure about and hands the rest on, and the bill is then set by how much gets handed on. That only works if the model's confidence means something, so it is measured before it is used.

Which is why the operating point comes from the table below rather than from the confidence number itself. Reviews are handed on in order of the model's own confidence, least sure first, and the threshold column is the lowest confidence still answered — the number that would go into a configuration file.

The first row is the reference: no deferral, the whole test set. That the score climbs as reviews are set aside is the evidence that the confidence ranks anything at all — had it stayed flat, no threshold built on it would work, whatever the calibration figure said.

And with the transformer behind it

The same ordering, but the reviews handed on are answered by HerBERT instead of being set aside. This is the row of the page that decides an actual system: the cheap model runs on everything, the expensive one on the fraction the cheap one is least sure of, and the GPU bill is that fraction rather than all of it.

Read the 20% row. Sending that share of reviews to the GPU — the 137 the baseline is least sure about — reaches 0.980 macro-F1, which is 85% of everything the transformer adds over the baseline, for 20% of the traffic it would otherwise have to serve. That rate is not picked for the number it produces: it is the smallest one in the table that closes at least 80% of the gap.

Two things this table is not. It is not a latency argument — the deferred reviews still wait for the transformer, so the tail gets slower, not faster. And the confidence it routes on is the baseline's own, which the calibration figure above says is not a probability; it works here because it ranks, and the ranking is what the first table measured.

Method, and what it cannot tell you

The configuration behind every number

Printed from the constants the code actually uses, not described. A setting that is quoted from config.py cannot drift away from the run that produced the figures above.

name        clarin-pl/polemo2-official
config      all_text
revision    802e35d2b12bae84bb07911d841e8f046dc2fcef
dropped     amb  (ambiguous — never merged into another class)
labels      negative, neutral, positive
license     CC BY-NC-SA 4.0 — downloaded on demand, never redistributed here

tfidf       ngram_range=(1, 2)  max_features=50000
            min_df=2  sublinear_tf=True
logreg      C=1.0  max_iter=1000  class_weight='balanced'
seed        42
fitted inside one Pipeline, so the vectorizer never sees the test split

model       allegro/herbert-base-cased
epochs      4  lr=2e-05  weight_decay=0.01
batch       4 x 4 accumulated = 16 per optimizer step
max_len     256
seed        42
the split batch is a memory accommodation, not a different hyperparameter: what
reaches the optimizer is the same 16 examples a larger card would pass in one go

bootstrap   2000 resamples of the test rows, 95% percentile interval
paired test exact McNemar on the reviews the two models answer differently
seed        42 — the interval is reproducible from the committed predictions

sizes       150, 300, 600, 1200, 2400, 4800, then the full corpus
seeds       0, 1, 2, 3, 4 per size
sampling    stratified — an unstratified draw of 150 rows can miss the neutral class outright

rates       0%, 5%, 10%, 20%, 30% of the test set, least confident first
calibration 10 equal-width confidence bins
probe floor 20 cases before a cell is read as a rate

python -m pl_review_sense.baseline_train   # trains, writes metrics + predictions
python -m pl_review_sense.analysis         # intervals, curve, probe, deferral
python -m pl_review_sense.site             # rebuilds this page into docs/

python -m pl_review_sense.herbert          # the GPU run behind the comparison
notebooks/herbert_colab.ipynb              # the same run on a free Colab card