Sign inRegisterCheck connection

Validating a Dayfile Parser Against a Fixture File

Validate a Cumulus dayfile parser against a pinned fixture: row count, max, min, and rain columns. Not a live /test/ debugger or current weather.

Back to Cumulus weather software guides

/test/dayfile.php looks like a parser you could hit in a browser to see whether dayfile.txt still loaded. That is the wrong place to validate a climate appendix. A public /test/ URL is a security smell: it often prints rows, paths, and PHP warnings from the live file. This article is the missing fixture test: a pinned dayfile, assertions on row count and on the max, min, and rain columns, and a fail when the contract drifts. It does not execute test PHP. It does not publish a fake month of weather as if it were current.

What a dayfile is—one row per meteorological day, extrema and accumulations, not realtime.txt—belongs on Cumulus dayfile.txt as a daily climate appendix. Moving a Cumulus 1 appendix into a fussier MX world is a migration problem, historically parked on /projects.phpdayfile. A second fixture for a forked column set is /test/dayfile1.php. This URL is the harness for one labeled file.

Historical context

The historical TNET PHP dayfile parser read a CSV-style appendix on a web host and printed monthly tables without a database. Operators then added a test script that pointed at whatever dayfile.txt FTP had just dropped. “It still prints” became the only regression test. When Cumulus added columns, or when a locale swapped comma and semicolon, the page still returned 200. The table was wrong.

The Cumulus Wiki dayfile page (last checked 13 August 2026) is the contract reference: one line per day, separator locale-dependent, field counts that grew as the software grew, MX stricter than Cumulus 1 about consistent dates, decimals, and widths. A test harness has to pin a generation of that contract. It cannot pin “whatever is on the station tonight.”

What the fixture is allowed to be

A fixture is a file you chose, sanitized, and stored next to the parser tests. It is historical or synthetic. It is not this afternoon’s observations.

Sanitize before the file leaves the station computer: replace the real location if it appears in comments, drop credentials, keep the column layout you need. Name the file so the generation is visible, for example dayfile-c1-core-max-min-rain.txt, not sample.txt.

Do not fill the fixture with invented “current” temperatures presented as a live station. If you need a known max of 10 and a known min of 2 for an assertion, label the row as a constructed example in the file header and in the test name. The assertion is about the parser, not about the climate of a town.

Assertions that catch real parser bugs

Row count, extrema, and rain are the three checks that fail when a dayfile parser silently shifts columns.

Row count. Count data lines after you have decided what a line is (watch for blank tails and missing final newlines). Assert equality with the fixture’s documented day count. If the fixture is a 28-row February example, a parser that reports 31 has invented days or split a quoted field. If it reports 27, it dropped a row. Official daily archives flag missing days; GHCN-Daily is built on that discipline (last checked 13 August 2026). A hobby parser that treats “number of lines” as “days in the month” fails the same way when a row is absent.

Date uniqueness and order. Assert that dates parse with the fixture’s documented format, that they are unique, and that they are in the order the file actually uses. Cumulus expects ascending dates. A merged line (missing newline) shows up as a bizarre field count on one row and a missing calendar day on another.

Maximum and minimum temperature columns. Bind by the documented field index for that generation, not by “the first number that looks like a temperature.” Assert:

  • the max column is greater than or equal to the min column on every complete row;
  • missing extrema are a missing token, not 0;
  • times of extrema, when present, are clocks, not leftover rain totals.

Zero is a legal Celsius reading and an illegal stand-in for “sensor was down.” The parser must not pad short rows with zeros to reach a later field count. Short rows belong to a different schema; that is the dayfile1 problem.

Rain column. Rain is an accumulation. Assert:

  • the rain field is the documented rain-total column for that generation, not rain rate and not a wind-run field that happens to sit nearby;
  • a missing rain value is not 0.0 unless the fixture explicitly records a dry day;
  • the unit is the station unit recorded in the test metadata, not guessed from the number’s magnitude.

A parser that maps Cumulus realtime field numbers onto a dayfile row will pass a naive “there is a number in column 6” check and still publish garbage. Index by the dayfile contract (Wiki field list), and store that generation in the test’s README.

Conceptual shape, not recovered TNET code:

$rows = parse_dayfile($fixture);
assert(count($rows) === 28);
assert($rows[0]['temp_max'] >= $rows[0]['temp_min']);
assert($rows[0]['rain_state'] !== 'missing' || $rows[0]['rain'] === null);

The third line is the scientific rule: missing and dry are different.

Pin the file; do not parse production in the test

If the test opens the live uploaded dayfile.txt, every end-of-day append changes the goldens. You will either rewrite expected values daily or stop looking at failures. Both outcomes bless drift.

When you upgrade Cumulus, capture a new fixture once, run the suite, and expect failures. If the new layout is intentional, update parser and goldens in the same change so the diff is the new contract. Keep the old fixture until you drop that generation. Operators do not upgrade in lockstep. Cumulus 1 accepted lines with 15 to 45 fields; MX releases expect a fixed width per build and read the whole file into an array (Wiki, last checked 13 August 2026). One fixture cannot honestly represent both. That is why a second test URL exists.

Do not use the live packet log as a substitute. A realtime log is a high-frequency series. Summing it is not a dayfile row. The harness should refuse to open realtime.txt when the function under test is parse_dayfile.

No live debug endpoint

Patterns that stay off the public web:

  • a CLI that exits nonzero on assertion failure;
  • CI on every parser change, with only sanitized fixtures in the repository;
  • a private operator status that says “last dayfile parse OK, row count N, file age,” without dumping rows.

Do not add /test/dayfile.php?pretty=1. Do not accept a filename from the query string. If you inherit that script, delete it or block /test/. The Cumulus hub catalogs explainers that replaced those endpoints.

Modern relevance

Monthly tables are still derived from historical daily rows. Whether those rows are usable depends on completeness, units, and schema identity—the same questions TNET states for public records in data sources, quality controls, and methodology. Distinguishing observed extrema from derived monthly sums is the vocabulary on how the service works. This page does not describe TNET internals and does not certify a hobby dayfile as an official climate product. For official climate, use NCEI. For warnings, use NWS.

Sources