Sign inRegisterCheck connection

Accumulating realtime.txt Snapshots into a High-Frequency Log

Accumulating Cumulus realtime.txt snapshots into a high-frequency log for later analysis: sample interval versus file growth, distinct from log rotation.

Back to Cumulus weather software guides

realtime.txt is overwritten. If you want a high-frequency series for graphs or later analysis, something must copy each new packet onto the end of another file. That homemade archive is a new dataset. It is not the Cumulus monthly log, not the dayfile, and not the live packet.

The historical TNET RealTimeLog script did that accumulation on a web host (PHP plus a scheduler). Independently, the Cumulus Wiki Toolbox documents a Windows-side running log of realtime.txt—for example, keep a line every minute even if Cumulus rebuilds the packet every 15 seconds—and names JPGraph as a consumer. This article is about sample interval versus file growth, and about labeling the result as derived. Rotation and retention are a different URL: realtime log rotation.

Modern TNET Weather does not ship the accumulator and does not maintain Cumulus.

Historical context

The Cumulus Wiki realtime.txt page states the properties that force this extra file into existence: optional, off by default, one line, rebuilt often, uploaded for near-current readings, not a historic series in the classic field list. Banners and heartbeats want that one line. A 24-hour temperature trace wants thousands of lines.

Two accumulation places were common.

On the web host. A cron job or scheduled HTTP hit ran PHP that read the uploaded realtime.txt and appended a timestamped copy to realtimelog.txt (the filename varied). This required the packet to arrive first. If FTP lagged, the log recorded late copies of the same line or skipped intervals. The server-check article is the quality-control sibling: it asks whether the latest line is fresh; this article asks how to keep the fresh lines.

On the logger PC. Toolbox’s realtime tab copied the local packet on its own interval and could upload the growing log. That path existed specifically for operators whose shared host had no cron. The Wiki’s wording is the scientific hint: Cumulus may produce the file every 15 seconds; the log may keep every minute. Those two intervals are independent instruments.

The TNET scripts catalog lists “Cumulus RealTimeLog Script” as a named package. This hostname does not rehost it.

Sample interval versus file growth

Every append is a choice about information and about disks.

Faster sampling can capture a frontal passage, a rain-rate spike, or a pressure plunge that a five-minute monthly log would flatten. It cannot invent samples faster than the hardware. The Cumulus Wiki notes that some EasyWeather-class stations update on the order of tens of seconds; a “real-time” file interval faster than the hardware is mostly cosmetic. Logging faster than the station’s native period duplicates the last value and pretends to be resolution.

Slower sampling writes fewer bytes and still supports a 24-hour plot if the phenomenon of interest is synoptic, not a one-minute gust. It will miss the gust. That is not a bug if the stated purpose is a temperature curve. It is a bug if the stated purpose is peak wind.

File growth is linear in interval inverse and in field width. A one-second append of a 50-field line, 24 hours a day, is a different operational object from a one-minute append. Unbounded growth fills disks, slows PHP that reads the whole file to draw a graph, and eventually fails the experiment. That failure mode is why rotation exists as a separate project. This page stops at: choose an interval on purpose, write it in the station notes, and do not treat “as fast as possible” as a scientific setting.

Do not pad this paragraph with a made-up megabyte-per-day figure. Measure the line length on your file and multiply by your chosen cadence if you need a budget. The relationship is arithmetic, not a TNET statistic.

What the homemade log is

Label the evidence.

  • Each source packet, if fresh, is observed (plus software-derived fields such as today’s extremes that Cumulus already computed).
  • The growing log is derived: an operator-chosen subsample, with the operator’s clock policy and the operator’s failure modes (duplicate lines, delayed FTP, cron gaps).
  • It is historical the moment a newer line exists.
  • It is not dayfile.txt. Daily max/min/rain belong in the dayfile appendix. You cannot reconstruct a true daily maximum from a log that sampled every ten minutes unless you accept underestimation.
  • It is not the vendor standard monthly log. That file has its own interval, columns, and rotation-by-month inside Cumulus. Citing a homemade realtime archive as “the Cumulus log” is a provenance error.

Clock policy belongs in the log, not only in the filesystem. Append the packet’s date and time fields, or a server timestamp you document, or both. File order after a crash is not a clock. Timezone and daylight-saving steps should be explicit or you will plot a two-hour phantom front.

Duplicates are normal when the uploader is faster than the logger. A later analysis should collapse identical packet clocks rather than treat them as extra samples. Gaps are normal when cron skips. A later JPGraph figure should lift the pen, not average across the hole.

Accumulation is not rotation

Accumulation answers: how do I keep the next line?

Rotation answers: how do I stop this file from becoming the only copy of last year, how do I name closed files so January is findable, and how do I avoid deleting the week that held the annual gust?

Mixing the two in one script is how people “fix disk full” by truncating the file at midnight and discovering in February that January’s extremes lived only in the deleted bytes. The unique path projects.realtimelogrotate is the retention article. This slash path stays on the append.

Practical checklist

  1. State two intervals: Cumulus realtime generation, and log append. They may differ.
  2. Do not log faster than the station samples. Cosmetic duplicates are not extra climate.
  3. Write a header or sidecar with Cumulus version, field map, units, timezone, and start date. A naked dump of lines is not self-describing.
  4. Treat FTP delay as a filter. You are logging what arrived on the host, not necessarily what the logger held.
  5. Plan rotation before the first full month. Accumulation without retention is a time bomb; retention without accumulation is an empty folder.
  6. Keep the live realtime.txt as the current packet. Do not point banners at the growing log.

Modern relevance

MX JSON and HTTP interfaces can reduce the need to scrape a one-line text file, but any current-conditions document that is overwritten still needs an accumulator if you want a series the vendor logs do not provide. A Pi that appends JSON snapshots to daily files is the same experiment with a different payload. Interval versus growth did not become obsolete.

Researchers who later find a realtimelog.txt on a backup disk should read it as an operator-derived series. It can be valuable. It is not automatically the station’s climate of record.

TNET research bridge

Building a series from overwritten snapshots is a data-source problem: cadence, completeness, and identity of the derived file. TNET’s data sources, quality controls, and methodology note is the modern discussion of those questions for records used in connection research. This page does not describe TNET internals.

Related pages: the projects map, JPGraph figures, the dayfile appendix, and log rotation. The Cumulus legacy hub lists the cluster.

Sources