pl-jobs-lora

A QLoRA fine-tune of a small Polish LLM that turns Polish IT job-posting prose into structured JSON — compared honestly against zero-shot and few-shot API baselines on accuracy, cost, and latency.

Status: in progress — baselines measured, fine-tune pending

The pipeline is built and tested end to end: the extraction contract, the prose→JSON dataset (leakage-guarded, split by publication date), the triangulated labeling-QA loop, the shared evaluation harness, and the QLoRA trainer. Four of six table rows are real measurements — the untuned base on local CPU and a frontier API baseline, both shot modes. What remains is the fine-tune itself, which needs a GPU outside this machine; its two rows are shown empty rather than estimated.

Evaluation

Every variant is scored by the same model-free harness on the same frozen 142-posting test set, per field. Both sides are normalized through the vendored it-job-radar functions, so paraphrases (ReactJS vs react) are not penalized.

Variant	JSON valid	Field F1	95 % CI	Seniority F1	Tech F1	$ / 1k	p50 s
few-shot API (claude-haiku-4-5)	1.00	0.51	0.47–0.54	0.62	0.28	4.47	2.3
zero-shot API (claude-haiku-4-5)	1.00	0.43	0.39–0.47	0.44	0.26	3.34	2.5
base, few-shot (Bielik-1.5B, CPU)	0.80	0.30	0.27–0.34	0.25	0.12	–	21.4
base, zero-shot (Bielik-1.5B, CPU)	0.05	0.04	0.01–0.07	0.07	0.01	–	67.9
QLoRA (ours)	–	–	–	–	–	–	–

Intervals are 95 % percentile bands from 2000 seeded resamples of the test set. Because 142 records is a small sample, differences are also compared paired — every variant scored on the same drawn postings — so a gap is only called real when it survives resampling. Every gap above does, except the two API rows tying exactly on JSON validity.

Two honest caveats about the last two columns. The local rows are priced –, not $0: no marginal cost is recorded for them, so the cost comparison is argued rather than measured. And their latency is CPU time, so it is not comparable to the API's — the fine-tuned row will be timed on a GPU, and the useful contrast will be accuracy per dollar, not these two seconds figures side by side.

What the numbers say so far

The untuned base zero-shot emits valid JSON on 5 % of postings — it rambles past the object. Few-shot recovers most of that (0.80), which is the gap the fine-tune targets.
The frontier baseline is not near-perfect either: 0.51 field F1. Part of that is a genuinely hard task and part of it is a ceiling in the data — see below.
Latency is where the local side already loses badly on CPU — 21 s versus 2.3 s per posting.

How much of the label is even in the input

The tech_optional field scored 0.01–0.02 for every model, the frontier baseline included. A metric on which the best available model does no better than the worst is usually not measuring the model — and it wasn't. Searching the postings directly, with no model involved:

Field	Gold terms	Present in the prose	Ceiling on recall	Postings where none appear
tech_expected	542	153	0.28	41 / 115 (36 %)
tech_optional	154	20	0.13	40 / 53 (75 %)

The gold technology labels come from the platform's structured widget, and the leakage guard strips that widget out of the prose — otherwise the task would be copying a list rather than reading a posting. In three quarters of the postings that have "nice to have" technologies, not one of them appears in the text the model is given.

This changes how the headline reads. The ceiling on tech_expected recall is 0.28 and the best model reaches 0.27 — 94 % of what the input makes recoverable. Read without the ceiling, that column looks like a weak model; read with it, the headroom is mostly not there to be taken. tech_optional was therefore dropped from the averaged headline — still scored and reported, but no longer treated as evidence about a model.

It is a bound, not a target: a term being present is necessary for extraction, not sufficient. Matching is deliberately conservative, so the real ceiling is somewhat higher — read these as a floor. It also sets expectations for the fine-tune: the open ground is JSON validity, seniority and work mode; the technology fields are near a data ceiling no fine-tune can lift.

Correction: the salary numbers published before 2026-08-18 were wrong

The scorer credited a prediction for matching None against None, and 107 of the 142 gold records carry no salary. A model that emitted nothing therefore inherited that base rate. The published table showed the zero-shot base at 0.75 / 0.87 / 0.74 on salary while it produced valid JSON 4.9 % of the time — that column was measuring how often postings omit pay, not accuracy.

Salary is now split into a detection decision (present vs absent, over all records) and value accuracy over only the 35 records that actually have a salary. The corrected picture is far worse and far more useful: even the frontier baseline recovers the currency on 23 % of the salaries present, the arrangement on 11 %, and the amount bounds on 6 %. Salary is the weakest part of this task by a wide margin, and the old metric hid that completely.

The same defect reached the base-model probe, whose ADR reported 0.74 / 0.96 / 0.74 on salary for a variant emitting valid JSON 0 % of the time. Both tables were regenerated from stored predictions; the base-model decision itself was unaffected. The correction is published rather than quietly restated, because the point of the project is the methodology.

Method