Quality checks

A quality check takes a random sample of a tagging session, lets reviewers check it with the keyboard and shows how well maesense and each labeler tag, with an interval of accuracy. Corrections of the reviewers reach your data only after someone decides on the lot.

The Quality section is available on request: it appears in the sidebar once the Quality product is enabled for your company.

Four steps

A check goes through four steps, and the run page shows them as the four cells of its summary strip. One step is open at a time; the other three show their result in a line. Click a cell to open that step.

1. Sample. The random sample is fixed when the check is created: how many messages of each executor are checked, by which plan. Nothing in the session changes until a lot is decided.
2. Review. Reviewers go through the sample with the keyboard. The step shows how far each lot is, who is checking, what the rules flagged and what is disputed. Review on the right of the header opens the next of your items.
3. Decision. Each lot is accepted or not: by the ISO 2859-1 plan, or by a person when the sample is not per executor. The table shows the defects found against the allowed number; a lot whose defects already reach the rejection number is marked “by the plan: not accepted” while the rest is checked. For a lot not accepted you choose the outcome here.
4. Report. Defects and accuracy per slice with a 95% interval, preliminary until every item is checked. The details (sentiment confusion, topics, error categories) are in the collapsed row under the table.

The pills above the steps filter every step by lot. Terms of the standard (code letter, AQL, Ac / Re) stay behind the “?” marks and in Lot acceptance.

Runs and lots

• A run is one check of one session. It is created from the session workbench with Check quality and takes the current filters of the workbench: the population is every message under these filters.
• Messages are split into lots by who produced their tags (the executor): maesense with a given model (for example “maesense · gpt-4o-mini”), or the person who last edited the tags. Messages edited by hand before authors were recorded form the lot “Manual edit, author unknown”: it is checked and shown in the report, but no decision is taken on it.
• Confirming a message without changing it does not change its executor: it stays with maesense. The report shows confirmed and unconfirmed messages separately.
• The sample is fixed when the run is created. Later edits or deletions in the session do not change it, and reviewers always check the tags as they were at that moment. A message changed since then is marked “Changed after sampling”.

Sampling

• Per executor (default): a random sample in every lot, because the decision is taken per lot.
• Simple random: one random sample of all selected messages.
• Stratified: in every lot the sample is split between strata (executor sentiment, platform or publication week) in proportion to their size.

The sample size comes from the margin of error you choose. By default it is ±10% at 95% confidence, with a correction for the size of the lot:

The creation screen also shows ±5%, ±7% and “Check everything”. A run holds at most 5,000 items. The run keeps its seed: the same seed and filters give the same sample.

Reviewing with the keyboard

Review on the run page opens the keyboard mode of the workbench in review mode. The form starts from the executor's tags; changing them builds your correction, and the message itself does not change yet. Below the form you see the discrepancies and the defect class.

• Space: without changes, “correct” and next; with changes, save the correction and next. Tags that are not in the instruction are marked in the header and are not in the form: Space saves the tags as the form shows them, so they are removed and the verdict is a correction.
• N: the next unchecked item. U: the next item flagged by the rules.
• E: error category. M: comment. X: disputed, pass the item to an expert.
• The other keys work as in keyboard mode: arrows, digits for sentiment, / for a topic, Esc to go back.

Each reviewer takes items in batches of 20 for 30 minutes, so two reviewers never get the same item. You cannot check your own work: items of your own lot and messages you edited or confirmed are never given to you.

Defect classes

A reviewer does not score by hand: the class follows from the difference between the executor's tags and the correction.

The class of an item is the worst of its discrepancies.

Disputed items

X passes an item to an expert, with your verdict kept. A user with the right to resolve disputes opens Resolve disputed on the run page; the expert's verdict becomes the final one. An expert never gets an item they reviewed or executed.

Rules

When a run is created, simple rules check every message of the population: no sentiment, more than one sentiment, a tag that is not in the instruction, an untagged message. Rules help reviewers find suspicious items (U), but the quality estimate uses only the reviewers' verdicts on the random sample.

Deciding on a lot

• With a per-executor sample, the lot is usually decided by its ISO 2859-1 plan, automatically, as soon as the sample is checked: see Lot acceptance. The steps below apply when acceptance is off or the sample is drawn another way.
• A lot is ready for a decision when every item has a final verdict.
• An editor or the owner opens the lot, sees the share of defects and minor defects with their intervals and chooses Accept or Do not accept, optionally with a comment. Nobody decides on their own lot. If the decider also checked items of the lot, for example in a team of two, the lot is marked “decided by a reviewer”.
• After the decision, accepted or not, the corrections of the checked items are written into the session in the background. The reviewer becomes the author of the new tags, and the edit history marks them as a quality check correction.
• A message whose tags changed after sampling is not overwritten: it is listed under Conflicts on the lot page with a link to the workbench, so you can decide by hand.
• An open run can be cancelled before any lot is decided. Its verdicts stay for the record; no correction reaches the data.

Reading the report

• The run page shows the whole run, maesense (all system lots), confirmed and unconfirmed messages of maesense, and every lot. Open a lot for its own report and decision.
• Shares are shown with a 95% interval: “87% (81–91%)” means the true share is most likely between 81% and 91%. The fewer items checked, the wider the interval.
• Each lot (and stratum) counts by its size in the population, not by how many of its items were checked. A small lot checked almost in full does not outweigh a large one.
• “Covers 66%” means part of the population has no checked items yet; the estimate is about the covered part only.
• When every message was checked, the share is exact and has no interval.
• Topics show precision (how many of the executor's topics are right) and recall (how many of the right topics the executor found). Topics with fewer than 10 cases are marked “few data”.
• Export CSV downloads every item with the executor's tags, the correction, the discrepancies, the class, the category, the comment and the reviewer.

Roles

• Every role sees runs and reports and can export the CSV.
• Reviewers, editors and the owner check items. Editors and the owner also create and cancel runs, resolve disputed items and decide on lots.

Details: Roles and permissions, Tagging workbench.