Rowfettle: Dataset Integrity Auditor — inspect the data before trusting the model

Introduction

The shortest route to an unreliable conclusion is to treat a successful import as evidence of a sound dataset. A table can be perfectly rectangular and still contain duplicate transactions, missing measurements disguised as zeros, or a category split by spelling. A chart may look plausible because the plotting library silently ignored invalid values. The program runs; the meaning has changed.

Rowfettle is a browser tool for that first review. It surfaces observable structure, explains the rules behind its warnings and lets you preview deliberate repairs on a copy. Begin with synthetic examples, compare the observations with the underlying records and decide what additional source knowledge you need. The useful output is a documented question or decision, not a cleanliness badge.

A useful audit begins with the unit of observation. What does a row represent, and what makes it different from another row? Write that answer before changing a cell. In practice, analysts often find that an important review question is an undocumented convention: an identifier imported as a measurement, a missing code treated as a category, or a timestamp whose zone was lost. This article teaches an inspection workflow using original synthetic fixtures, with explicit limits and no claim that the tool can establish truth from patterns alone.

Types of data integrity problems

Completeness, representation, uniqueness, plausibility and relationships are separate concerns. A missing salary and a recorded salary of zero describe different evidence. A repeated laboratory observation may be valid, while an identical import of a transaction may count revenue twice. A country code and country name can form a legitimate one-to-one mapping without either being an erroneous column.

Survey data often exposes category normalisation and nonresponse. Sensor data adds order, clock regularity and calibration. Financial records need transaction identity and sign conventions. Laboratory tables require units and censoring semantics. Inventory lists need explicit expectations for IDs. Domain presets tune the questions; they do not invent a schema or certify a domain.

Completeness and representation

Completeness is relative to a declared schema. A blank response, a text missing token and a recorded zero should remain distinguishable until the source definition says otherwise. Representation includes delimiter choice, Unicode, date zones and numbers embedded in text. The strict numeric subset excludes unit suffixes. That exclusion protects meaning: a laboratory value carrying a unit should be converted deliberately before it becomes a bare number.

Identity and relationships

Uniqueness concerns identity under a declared key. Exact tuple equality can confirm a copied record, but the source might contain two legitimate identical events. Normalisation ignores selected surface distinctions; fuzzy similarity proposes additional candidates. Column relationships raise a separate question: are the variables independent measurements, redundant encodings, or derived information unavailable at prediction time? A correlation alone cannot answer that question.

Plausibility and order

Plausibility depends on context. Distribution fences identify unusual observations, while adjacent-record checks describe the ordering you supplied. A clock gap could reflect an outage, a changed schedule or a reordered extract. The inspection result should preserve those alternatives until source evidence resolves them. No automatic repair follows a warning.

Reusable data checks

A reusable rule set turns a question into a repeatable check. For a survey, require an identifier and constrain age to the range justified by the study protocol. For a ledger, compare a total against its named parts with an explicit absolute tolerance. For a study timeline, check that a valid ISO start date is no later than its end date. These are user-declared expectations; passing them does not establish scientific validity.

The tool imports a bounded version-1 JSON rule set or a Suite v2 document (and the supported subset of Table Schema, CSVW and expectation-suite files), exports Suite v2, and rereads every source row when you select Run rules, including in streaming profiling mode. Required, inclusive range, exact allowed-value, date-order, sum, length, pattern, mask, type, comparison, uniqueness and reference checks return separate passed, failed and skipped counts, and an imported item the tool cannot evaluate is reported as not checked rather than passed. Missing operands skip comparisons; invalid present operands fail. A rule referring to an absent column refuses the pass. Up to eight failed data-row examples help you inspect source context.

The missingness pattern matrix reports joint absence across the first 16 columns. It retains the first 32 distinct masks with exact counts and groups every additional pattern into Other rows. It describes observations under your missing-token policy; it cannot distinguish missing completely at random from a sampling or collection problem.

Validation JSON and Markdown exports include a SHA-256 of the stated validation payload. Repeating the same checks with the same results reproduces that digest. Different source files can produce identical validation payloads, so the digest is not source-file identity, an authenticity signature or a replacement for retaining the source and application revision. The existing full profiling text report remains a separate export. Keep the original source and the exported payload together.

Portable contracts: a subset, stated as a subset

A Suite v2 document can be exported as a Table Schema document and imported from Table Schema, CSVW metadata or an expectation-suite file. In each direction only the supported subset moves: declared columns with their type, required, unique, length, pattern, range and enumeration constraints, and the primary key. Anything outside that subset is listed by name as not checked, never silently dropped and never counted as a pass; an export that would lose semantics refuses instead of writing a weaker contract. The wording is deliberate: the tool reads and writes a subset of those formats. It does not claim to be compliant with, certified for or a complete implementation of any of them, because no conformance suite has been run and none is bundled.

Input inference with evidence

Type inference reports a confidence as a fraction with a visible denominator: the number of present values that parsed as the inferred type over all present values, with the alternatives considered. Numbers carry the decimal and grouping grammar that was used; identifiers with leading zeros are flagged so that a number override, which would drop those zeros, is a deliberate choice. Ambiguous slash dates are reported as ambiguous and stay text rather than being resolved silently. Every inferred type has an override, and an override takes effect only after Analyze runs again, so the displayed statistics always correspond to the displayed type.

Profile and missingness with denominators

The column profile lists quantiles, moments, counts, text lengths and the date range with the count each number is based on: rows for present and missing, present values for zeros, negatives and lengths, numeric values for quantiles and moments. Skewness and excess kurtosis are labelled population moments and read Unavailable below three or four observations or at zero spread. The missingness matrix covers up to 64 user-selected columns as exact pattern counts with an explicit Other row; a reconciliation line shows that pattern rows plus other rows equal all rows. Phi coefficients and conditional means describe association between absences. None of these outputs names a missing-data mechanism, because a table cannot establish why a value is absent.

Recipes: ordered steps, undo, redo and replay

Every repair is a step in an ordered recipe: imputation, disposition, row removal, exact-duplicate removal, explicit value maps, renames, whitespace and case changes, replacement, number parsing under a declared grammar and date rewriting. Undo removes the last step from the export and redo restores it; a new step after an undo discards the undone tail. The preview and the export run the same materialised plan, so the counts you read before exporting are the counts the export produces. A recipe can be saved as JSON and replayed on a second compatible file: each step is bound to its column by exact name, unbound steps are named and the export is refused until they are removed. The change log of a replay starts with the recipe digest and the binding record, and the exhaustive JSON Lines history names the step behind every changed cell.

Record linkage as similarity, not certainty

Record linkage compares rows of the analysed file with each other under a configuration you declare: match fields with comparators and weights, blocking rules that decide which rows are compared at all, and a match threshold. The score of a pair is the weighted mean of its field similarities, a number from 0 to 1. It is a similarity under the declared comparators, not a measure of how sure the tool is that two rows describe one entity. Every count carries its denominator: candidate pairs formed by blocking, comparisons made against the comparison budget, pairs at or above the threshold, and every block that exceeded its cap with its key and size. When a truth column is supplied it is excluded from scoring and blocking and used only to compute blocking recall and precision, recall and F1 across a threshold sweep, each shown with the counts behind it. Accepting a pair adds a row-removal step to the recipe; nothing is merged. Agreement of the evaluation with an independent enumeration on a labelled public corpus is not yet measured.

Key parameters

ParameterMeaningConsequence
Missing tokensYour definitions of absent evidenceChanges denominators and which cells can be imputed
Type overrideNumber, text, date or inferredPrevents numeric-looking identifiers from being analysed as quantities
Domain presetContext for extra findingsAdds clock, finance, lab or sequence questions
SensitivityConservative to loose fuzzy matchingChanges candidate thresholds, never establishes identity
Memory envelopeAdmission byte counters plus capped retained stateHelps select exact or streaming analysis
Expected stepSafe-integer sequence spacingDefines absent expected positions
View range/logHistogram presentationChanges the chart without changing the input statistics

Check these settings against the data dictionary before interpreting any count. A default is a convention, not domain knowledge.

Key formulas and register IDs

For numeric observations, the mean is sum(x)/n. Welford updates the mean and centred squared sum as observations arrive, then sample variance divides that sum by n−1. The tool refuses nonfinite numeric encodings and aggregate overflow rather than manufacturing a statistic.

Exact quantiles linearly interpolate sorted position (n−1)p. The interquartile range is Q3−Q1; the displayed excursion rule flags values strictly below Q1−1.5 IQR or above Q3+1.5 IQR. Median absolute deviation is the median of absolute distances from the median. These conventions describe a sample and do not prove that an observation is erroneous.

Pearson correlation divides centred cross-products by the two centred lengths. Spearman applies correlation to average ranks, which can reveal a deterministic monotonic curve. Streaming quantiles use a per-numeric-column KLL-style compactor with a conservative deterministic rank bound. This bounds rank discrepancy rather than a fixed error in measurement units.

Mean and sample spread (F01–F02): mean = sum(x)/n and sample SD = sqrt(sum((x−mean)²)/(n−1)). Here x denotes a finite observed numeric value and n denotes the numeric count, rather than the entire row count. Use these summaries for a quantitative column; identifiers that happen to contain digits should be given a text override. A lone observation has no sample SD.

Quartiles and dispersion (F03–F04, F07): the exact quantile at probability p uses sorted position (n−1)p with interpolation. IQR = Q3−Q1. MAD = median(abs(x−median(x))). Fences use the displayed quartile convention; changing conventions can change which boundary observations are flagged. Fence membership is strictly outside, so equality with a fence remains inside.

Missingness and duplicate excess (F05–F06): missing fraction divides the selected missing-cell count by the number of cells eligible for that display. Duplicate excess counts copies after the first identical tuple. Both are empirical counts under the selected conventions; neither infers whether a record ought to exist.

Relationships (F10–F12): Pearson divides centred cross-products by the product of centred lengths. Spearman repeats that calculation on average ranks, including ties. An affine hint uses the same paired observations for its slope and intercept. Pair counts matter when either column is missing. Comparing an all-column denominator with a paired statistic can conceal exclusions.

Streaming rank discrepancy (F22): every KLL compaction contributes at most its input weight to the rank-discrepancy ledger. The displayed fraction divides the accumulated ledger by the column's numeric count. This conservative deterministic bound is independent of the row reservoir. Fixed-bin histogram mass can differ by at most the discrepancy at each of its boundaries; a derived outlier fence also carries threshold uncertainty. The complete output, source-status and independent-case mapping is in FORMULAS.md beside the engine. Published-source validation remains pending-source; reproducible mathematical verification is a separate category.

Learning objectives

After working through the tool, you should be able to distinguish missing evidence from a recorded zero; explain exact, normalised and fuzzy duplicates; calculate how an integer gap depends on the expected step; compare a histogram before and after an imputation; identify which streaming results use every row; and document why a repair was accepted or rejected.

A practical review also identifies what the tool cannot infer: collection history, legitimate repetitions, causal structure, missingness mechanism and when a target became available. Those questions belong with the people responsible for the data and the analysis design.

  1. Calculate missingness using the selected tokens and the correct denominator.
  2. Distinguish exact equality, normalised equality and a fuzzy candidate.
  3. Reproduce the survey mean from observed values rather than hidden generator parameters.
  4. Explain why an ID gap requires an expected sequence and a known observation window.
  5. Compare the original and remedy distributions on identical histogram edges.
  6. Document which results cover the full stream and which use bounded samples or sketches.

Exploration activities and worked case studies

Start with the Survey example and inspect its raw rows. Count the missing cells before reading the summary. Compare exact duplicate evidence with normalised label variants. Ask whether punctuation or case carries meaning before accepting a spelling cluster.

Next choose Sensor and inspect timestamp order. Use the row viewer around the clock gap and repeated timestamp. Explain why the first positive interval is an assumption about the schedule, and how a legitimate schedule change could trigger the same finding.

Then choose Transactions. Identify the exact repeated tuple and the negative amount. Decide whether the negative amount is a refund, a sign error or unknown; the tool cannot decide from the number alone. Preview removing the excess tuple and compare output rows with the original.

Finally use Entry review with the expected sequence step. Explain the absent positions without generating replacement records. Inspect the spelling transposition candidate. Change sensitivity and observe that candidates and accepted decisions are different things.

Activity: survey responses

The synthetic survey has 10 rows, 2 missing cells and 1 excess exact duplicate row. Its score mean is 26.25. The conspicuous high score warrants inspection, but an unusual score can still be valid. The label column includes case and spacing variants; normalisation groups them for review without changing the source.

Select score and preview median imputation. The missing count changes because the remedy fills the cells matching the missing-token definition. The histogram overlay shows a concentration at the chosen fill value: imputation has created repeated values, not recovered unknown observations. If missingness depends on the score, a convenient fill can bias a later result.

Now reset remedies and instead flag the column. This preserves raw evidence while recording a decision to avoid using it pending review. The distinction matters when downstream users see a cleaned export: a value derived from a rule should not masquerade as a fresh measurement.

Activity: sensor observations

The sensor fixture has 6 rows. Its timestamp column produces 1 clock-gap interval and 1 adjacent repeated timestamp; reading has 2 flat-line transitions. These counts follow explicit adjacent-record definitions, not a diagnosis of a failed device. A repeated timestamp might identify two sensors at the same instant if the sensor identifier was omitted.

Forward fill can be reasonable for some state variables, but it can conceal an outage or fabricate a measurement stream. Backward fill uses later observations and can leak future information. Linear interpolation here uses row position rather than elapsed time, so unequal intervals make it unsuitable without deliberate resampling. Inspect source timing and domain semantics first.

Activity: transactions

The transaction fixture has 6 rows, 1 excess exact duplicate, 1 negative amount and 1 missing amount. Identical tuples suggest import overlap, but transaction identity may require a source-system key. Removing a legitimate repeated sale would undercount activity just as retaining an import duplicate can overcount it.

The negative amount is labelled as a refund in the fixture. Its sign therefore prompts a review rather than an automatic correction. First-digit and round-number findings are only screening hints. Assigned prices and constrained ranges often violate the assumptions needed for a meaningful Benford comparison. The tool makes no fraud claim.

Activity: sequence and repair decisions

Load Entry review and choose the sequence column. Inspect the absent expected positions, then seek source rows around an observed jump. Record why the expected step is appropriate. Change the near-duplicate sensitivity, inspect a proposed transposition and leave it unchecked until the source meaning is clear. Finally, select a remedy on a copy, preview it, and compare the changed-cell and removed-row totals with the exported history. The expected lesson is that a candidate, an accepted decision and a transformed value are different objects.

For each activity, save a short decision note: input convention, observed finding, source question, proposed action and reason. Keep rejected candidates in your reasoning too. Experienced reviewers find that documenting why a plausible merge was refused is often as valuable as recording accepted repairs. A concise note prevents the next analyst from repeating the same uncertain decision.

Real-world applications

  1. Survey operations: inspect nonresponse tokens and spelling variants before aggregating a category. Confirm whether punctuation or case encodes a meaningful response option. The survey fixture demonstrates representation review without assuming that every unusual score is wrong.
  2. Sensor collection: compare timestamp transitions with the acquisition schedule, inspect repeated instants and distinguish a steady reading from a recording outage. The sensor fixture supplies a reproducible clock-gap example; a deployment requires instrument logs and sensor identity as well.
  3. Financial reconciliation: investigate import overlap using source transaction keys and tuple evidence. A negative amount may represent a refund. Keep the financial preset as screening support; its digit and rounding hints do not establish fraud or accounting correctness.
  4. Laboratory ingestion: identify mixed unit suffixes and censored observations before numeric summaries. A detection-limit string describes an inequality, not a measured zero. Convert units under a reviewed protocol and preserve the original lexical evidence.
  5. Inventory maintenance: review absent or repeated safe-integer positions against the allocation policy. Some systems reserve or retire identifiers deliberately. Do not manufacture rows merely to make a sequence look consecutive.

Comparing spreadsheet and profiling workflows

A spreadsheet offers filters, conditional formatting, pivot tables, formulas and familiar row inspection. It can be a good place to answer a focused question when the import rules, ranges and formulas are controlled. Its flexibility also puts responsibility on the analyst to preserve raw data and document transformations. No speed, capacity or error-rate comparison was measured in this lane.

Python profiling libraries support programmable reports and integration into repeatable pipelines. Validation frameworks can encode explicit expectations and run them at ingestion boundaries. Interactive cleaning tools can support rich transformation histories. These are workflow categories, not a tested current feature ranking of particular packages.

This browser tool’s intended role is an immediate local review with visible conventions and explicit repair previews. For scheduled pipelines, modelling splits, governed schemas or extremely large data, choose a workflow whose operational guarantees match the task. A website summary does not replace a reproducible analysis pipeline.

For a recurring process, a versioned script or an explicit validation framework can preserve the input schema and transformation parameters with the data pipeline. A browser review is useful when a person needs immediate inspection, but a screenshot is not a machine-checkable ingestion contract. Choose the workflow by operational needs, reviewability and ownership. This article does not rank particular packages or assert comparative speed, capacity or detection accuracy.

Reference data: reproducible synthetic observations

These values come from the bundled examples and are asserted against actual engine runs by v2.test.ts. They describe invented fixtures rather than a population or an industry benchmark. The reference table deliberately contains observed results, not purported universal data-quality thresholds.

FixtureOutputObserved valueIndependent basis
SurveyData records10Enumerated synthetic tuples
SurveyMissing cells2Selected missing-token enumeration
SurveyScore mean26.25Observed score arithmetic
SurveyExcess exact duplicate1Raw tuple equality
SensorData records6Enumerated synthetic tuples
SensorClock-gap interval1Adjacent timestamp differences
SensorAdjacent timestamp repeat1Equal neighbouring instants
SensorReading flat-line transitions2Equal neighbouring numeric readings
TransactionsData records6Enumerated synthetic tuples
TransactionsExcess duplicate / negative amount / missing amount1 / 1 / 1Separate observed-record checks
Entry reviewRecords / absent integer positions20 / 2Tuple and expected-position enumeration

The table is a reproducibility anchor: load the named example and compare the output before inspecting private data. Agreement verifies that fixture under its declared settings. It does not validate a different source schema, missingness mechanism or browser's memory capacity. The guide links to captures taken from the actual running tool, with invented data only.

Challenge questions

  1. Conceptual: why can a table with no missing cells still contain serious integrity problems? Give a source convention that an automatic count cannot discover.
  2. Calculation: reproduce the survey's score mean from its present numeric observations. Explain why its numeric denominator differs from its row count, and what would change if a missing token were reclassified.
  3. Analysis: explain the sensor's clock-gap and repeated-timestamp counts from adjacent records. Could the same evidence arise from a legitimate schedule change or several sensors? State what additional source field you need.
  4. Application: the transactions fixture contains a negative amount and an exact repeated tuple. Design separate review decisions for the sign and the repeated record. Explain why a single “remove anomalies” action would lose provenance.
  5. Design: plan a large-file review whose exact storage does not fit the browser allowance. Identify the outputs that remain full-stream counts, the outputs that become estimates and the acceptance checks you would require before applying a repair downstream.

Discuss answers with the people responsible for the source. The calculation question has a fixture answer; the design question has several defensible answers because identity, collection policy and intended use are external to the observed table.

Common mistakes and deliberate remedies

A remedy is a modelling choice. Mean and median replace uncertainty with a single value; mode can erase rare categories; a constant can confuse absent evidence with a real zero. Forward and backward fill require meaningful order. Interpolation requires a defensible relationship between neighbouring observations. Dropping rows or columns changes the population being analysed.

Accept each spelling mapping explicitly, inspect row candidates and preview the complete plan. The tool reports changed cells, removed rows and selected-column before/after statistics. It never rewrites the original. The CSV export quotes cells and neutralises spreadsheet formula-leading characters; that protection changes lexical representation and needs consideration by a downstream machine parser. The exhaustive JSON Lines history records these protective header/cell events under csv-apostrophe-v1 separately from semantic repairs; completion totals distinguish both operation classes.

The on-screen change preview is bounded. The downloadable JSON Lines history streams every changed cell, removed row and column decision, followed by completion totals. An interrupted export without a completion record is incomplete. Save the original, cleaned copy and history together when provenance matters.

  • Treating missing as zero: this creates an observation that was never collected. Inspect the source definition and use a justified missingness model.
  • Equating similarity with identity: a plausible spelling merge can join distinct categories. Review every mapping and preserve the rejected alternative when its meaning is uncertain.
  • Using future information: backward fill and whole-dataset imputation can leak information across time or modelling splits. Apply a transformation in the correct analytical context.
  • Dropping valid extremes: an IQR excursion describes position in a distribution. Check source and domain evidence before removal.
  • Ignoring changed lexical output: spreadsheet-safe CSV prefixes formula-leading strings, including negative encodings. A downstream machine parser should explicitly handle that protective convention.

Exact mode imputation selects the most frequent original category before any display cap. Streaming mode is refused unless its global winner is certified from frequency bounds. When a streaming top-category list excludes long over-budget values, it does not certify a global mode. Leave the missing cells unchanged or establish the mode in an appropriate complete-data workflow. Likewise, a sampled relationship should never become an automatic column-drop decision.

Frequently asked questions

Does a high health score prove that the data is correct?

No. F19 defines an explicit heuristic from missingness, duplicate excess and serious facts; review suggestions contribute no penalty. It is not a calibrated probability, a collection-quality certification or a fitness-for-use verdict. A complete but systematically biased sample can score well.

Why can two tools give different quartiles?

Quantile conventions differ, especially for small samples. This tool declares interpolated sorted positions in F03. Compare conventions, missing-token rules and numeric eligibility before treating a disagreement as an arithmetic defect.

Does perfect correlation prove leakage?

No. A derived variable can be legitimate if it is available at the intended decision time. Conversely, an imperfect relationship can still leak information. Pearson and rank evidence identify patterns; provenance establishes availability and causal context.

Why does the streaming tier show error bounds?

Bounded sketches retain less information than a full exact column. KLL rank discrepancy and frequent-item intervals describe different kinds of uncertainty. Streaming HLL distinct estimation uses a calibration coefficient computed from its defining integral in the engine and checked against an independent reference; an optional override is recorded as such, and no portable standard error is asserted. Read each output's basis rather than applying a single percentage to the entire report.

Can the tool recover absent source records?

No. An integer gap or clock gap shows an absent expected position under a declared schedule. The tool cannot reconstruct the missing observation or establish that the collection process was intended to fill that position.

Can I use a native spreadsheet workbook (.xlsx)?

Native workbook parsing is deferred. Export the desired sheet to UTF-8 CSV, preserve identifiers and date semantics, and inspect the result. The application does not fetch an extra parser or offer an offline edition.

Does the tool send my file anywhere?

Your data stays on this device; dataset contents are not uploaded. Settings are saved in this browser; a recipe is saved only when you ask for it. Exports, including large exports, may use the browser’s private temporary storage for cleaned records and change history. For paired exports, use each file’s download button. Keep this tab open until you have downloaded both files. Pending downloads cannot be recovered after refresh; remove leftover temporary files and rerun the export. Temporary files stay until both downloads have started, you remove the files, or recovery reclaims them. Removal follows a short handoff delay; cancellation removes unfinished files. If the tab closes or an export is interrupted, files can remain until you remove them with “Remove leftover temporary files” or the next temporary-storage operation reclaims them. Temporary duplicate indices contain hashes and byte offsets.

The analysis code makes no network request with your data, and this is checked rather than asserted. A Chromium privacy matrix runs the complete workflow on every synthetic example and on a generated synthetic file (the harness default is 100 MB; each receipt records the size it ran) while recording every request's origin, method and body size; it passes only with zero body bytes, zero non-GET requests and zero requests to another origin. A separate source gate parses every engine file and refuses network identifiers. The site's shared gate fetches a run token with a GET whose query names the tool; it carries no dataset records.

Large-file controls and coverage

Exact storage uses typed numeric lanes, dictionary-encoded raw values and missing bitmaps. Streaming analysis uses bounded state, frequent-item estimates and per-column KLL compactors and a row reservoir. The full stream still contributes missingness, online moments, bounded Pearson cross-products and adjacent-entry evidence. Sketch-based quantiles and sample-based ranks, fuzzy checks and column mappings carry different coverage labels.

A small rank-error bound does not imply a small value error. Imagine a distribution split by a large gap: a narrow rank interval around the median can span that gap. Derived IQR fences and MAD have additional uncertainty. Exact distinct counts are enumerated. Streaming distinct estimates use the computed calibration coefficient and carry no supplied standard-error claim, while frequent-item counts report their own intervals.

Workers keep computation away from the page’s main thread, and pause/cancel operate at chunk boundaries. Memory guards can restart an exact attempt in streaming mode. Result-detail admission and complete report serialization also have byte allowances: dense relationships or repeated long names may require an explicit refusal even for a small source. No truncated report is labelled complete. Those mechanisms are engineering safeguards; shared-host synthetic receipts are directional and do not establish a capacity or responsiveness promise for every browser. Read the tutorial’s limits before planning a workload.

Sequential framing is necessary for quoted CSV newlines and nested JSON objects. Complete record-aligned buffers then go to a bounded parsing pool, using transferable buffers rather than copying the entire file into the page. Results drain in input order so timestamp, entry and seek evidence remains reproducible. Numeric sorting also occurs away from the page. The parsing pool can shrink under a small allowance even when the machine reports many hardware threads.

Backward/linear repairs seek original source anchors under bounded caches, and streaming duplicate removal confirms candidates through transient hash/offset partitions. If temporary storage is unavailable, exact prefix rereads preserve correctness at the cost of additional I/O. Pause and cancel remain available; an interrupted export is incomplete.

Exact duplicate-row counts complete through the same hash/offset partitions on streamed files; a hash only nominates candidates and the original records are reread and compared before anything is counted. Where temporary storage exists, normalised duplicate counts complete in a separate hash-only pass after the exact pass; without it they remain labelled lower bounds. Partitions store hashes and byte offsets only, never cell text, and are removed on completion or cancellation. The largest file size at which these exact counts have been checked against planted truth in Chromium is not yet measured.

The Memory envelope control determines a smaller admission allowance for retained analysis state. The allowance covers column dictionaries, duplicate candidates, bounded frequent items, sketches, run sets, pair/rank scratch, parser messages and output buffers conservatively. Retained analysis state is capped by admission counters; they do not measure heap usage or bound transient peaks. Admission counters are not working-set measurements. Total browser-process memory is not guaranteed; runtime and garbage-collector overhead vary. A diverse long-cell fixture is necessary because a repeated-value large file alone does not stress dictionary cardinality. The retained engineering receipts explicitly distinguish these cases.

Privacy and about the data

The analysis code processes local files in browser workers and sends no dataset records to a server. The shared site gate obtains a run token without dataset content. Only allowlisted settings are autosaved; datasets, file names, findings and repair mappings are not. Exported copies and reports are files you control and may contain sensitive information.

Native .xlsx workbooks, complete all-stream fuzzy searches, integer ranges beyond the run-set allowance and complete all-stream categorical dependencies are not claimed. JSON uses a first-record schema and one nested object level; UTF-8, cell and record limits are explicit.

All bundled datasets are original synthetic examples. They contain no real respondent, customer, account or laboratory record. Images in the tutorial are captures of those examples running in Chromium. Dataset contents, filenames, accepted mappings and selected rows are excluded from settings persistence. Temporary duplicate indices contain hashes and offsets only; they are removed on normal completion or cancellation. The export fallback also temporarily stores output data in private browser files under a lease. Completed files are downloaded by object URL and then removed; abandoned operations are recovered on the next storage operation. If private storage is unavailable, only bounded small exports can use memory. None of these facts makes downloaded reports or screenshots non-sensitive when you inspect your own data.

References and source status

The following are openly accessible source locations for methodological follow-up. These external pages have not been inspected, and their published examples remain awaiting source validation. A link is not evidence that a particular source was validated. The repository's independent mathematical and direct-enumeration checks are identified separately.

  • NIST Engineering Statistics Handbook: https://www.itl.nist.gov/div898/handbook/ — distribution summaries and inspection context.
  • Python statistics documentation: https://docs.python.org/3/library/statistics.html — independently executed standard-library reference functions.
  • R quantile documentation: https://stat.ethz.ch/R-manual/R-devel/library/stats/html/quantile.html — quantile-convention background, pending external inspection.
  • Welford, corrected sums of squares, DOI https://doi.org/10.1080/00401706.1962.10490022 — bibliographic algorithm provenance, pending source inspection.
  • Karnin, Lang and Liberty, Optimal Quantile Approximation in Streams: https://arxiv.org/abs/1603.05346 — KLL-family background; this implementation uses its own conservative compaction ledger rather than borrowing an unverified confidence constant.
  • HyperLogLog analysis: https://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf — estimator background, pending external inspection.
  • JSON format specification: https://www.rfc-editor.org/rfc/rfc8259 — format background; source-validation status remains pending.
  • CSV format reference: https://www.rfc-editor.org/rfc/rfc4180 — common quoting context; the tool's accepted subset is documented explicitly.
  • Unicode normalisation: https://unicode.org/reports/tr15/ — representation background, pending external inspection.

How to cite and continue learning

Cite “Simulations4All, Rowfettle, website” together with the date, exact/streaming tier, selected input conventions and retained report. Include the repository revision in technical work when available. Do not cite the health score as evidence that a dataset is scientifically valid. Reproducing the input and plan matters more than reproducing a screenshot alone.

Use the step-by-step tutorial, the Verify page, and related Factorlab and Solvegrid workflows when available to your account. They address different questions about experiments and decisions.

Verification log

The Verify page runs the ready checks and shows their inputs, expected results, tolerances and basis. Checks awaiting published-source validation are excluded from the pass count. Independent numerical comparisons and regression tests also cover selected calculations and parser boundaries. A passing case supports only the conditions it tests.

Browser checks in Chromium exercised the interface and worker, including guide and verification rendering, histogram labels, remedies and downloads. Synthetic data was used to check privacy and a scripted analysis task, including whether exported failing rows matched the planted failures. Firefox and WebKit were not checked in this work. These checks do not establish a file-size or performance guarantee for every device. General relevance of review suggestions to your data is not yet measured. The largest file size at which these exact counts have been checked against planted truth in Chromium is not yet measured.

Facts describe missing-token counts, exact duplicate rows and parser results under the stated settings. Inferred patterns appear under Worth reviewing with a neutral Suggestion label. These patterns can be valid in your data. Check them before changing anything. Review suggestions do not reduce the health score.

Rowfettle: inspect data before trusting the model | Simulations4All