Publishing a dataset at a stable URL

This page is for the team inside an agency that publishes a data file and replaces it at each release. It sets out a layout for the files and their URLs that keeps every link working, lets a program fetch the newest release without a person, and survives a change to the columns. It is the layout this site uses for every dataset it serves, and it comes from what breaks across the government sources the site reads. Nothing here needs a portal, a database or an API. A folder on an ordinary web server is enough.

Two URLs for every file

A dataset that changes needs two kinds of URL. A dated URL names one release and returns the same bytes for as long as the site exists. A current URL always returns the newest release and never changes. Each does a job the other cannot. A report cites the dated URL, so a reader years later opens the file the author used. A program reads the current URL, so it picks up each release without anyone editing it.

Most sites publish one or the other. A file that is overwritten in place gives a current URL and loses the history. A file with the release date in its name, such as enrolments_26052026.xlsx, gives a dated URL and breaks every link at the next release, because the new file has a new name and the old one is usually taken down.

The layout below gives both.

/data/senior-enrolments/
  index.html                        the dataset page
  latest/senior-enrolments.csv      the current URL, a redirect to the newest dated file
  latest/senior-enrolments.xlsx
  v/2026-05-26/senior-enrolments.csv
  v/2026-05-26/senior-enrolments.xlsx
  v/2026-05-26/schema.json
  v/2025-06-20/senior-enrolments.csv
  v/2025-06-20/senior-enrolments.xlsx
  v/2025-06-20/schema.json
  versions.json                     every release, newest first
  schema.json                       the current column list
  changes.md                        what changed at each release

The date goes in the path and the file name stays the same, so a saved file has the same name whichever release it came from, and the folder it sits in says when it was made. The date is the day of the release as YYYY-MM-DD, which sorts into order in any file listing.

The current URL

The current URL should redirect to the newest dated file with status 302 or 307, so that a program which follows it can see from the final address which release it received. If the web server cannot redirect, overwrite the file at the current URL with a copy of the newest release instead. The bytes change and the address does not, and a program can tell releases apart by the Last-Modified and ETag headers, so send both.

Publish versions.json beside it. It lists each release with its date, its dated URL, its SHA-256 hash, its row count and the schema version it follows. A program that reads this file can tell whether there is a new release without downloading one, and can fetch an older release when it needs it.

{
  "dataset": "senior-enrolments",
  "schema_version": "1.3",
  "versions": [
    {"date": "2026-05-26", "schema_version": "1.3", "rows": 48213,
     "sha256": "9f2c…", "url": "/data/senior-enrolments/v/2026-05-26/senior-enrolments.csv"},
    {"date": "2025-06-20", "schema_version": "1.2", "rows": 46990,
     "sha256": "41b0…", "url": "/data/senior-enrolments/v/2025-06-20/senior-enrolments.csv"}
  ]
}

The dated URL

A dated file is never edited and never removed. If a release turns out to be wrong, publish a corrected release under a new date and say in the change log which release it replaces and why. The wrong file stays where it is, because reports already cite it. Send Cache-Control: public, max-age=31536000, immutable on a dated file, so that browsers and proxies keep it, and a short max-age on the current URL, so that a new release is seen within minutes.

Headers and access

Serve each file with its correct Content-Type (text/csv; charset=utf-8 for CSV), a Content-Disposition header carrying the file name, Last-Modified, ETag, and support for HEAD and range requests, so a program can check for a new release cheaply. Nothing on the data path should need a login or JavaScript.

If the site runs a bot challenge, exempt the data paths from it. A challenge exists to stop programs, and the programs it stops include every one the data is published for. A program cannot pass a challenge and should not try, so a dataset behind one is read by hand or left alone. A rate limit on the data paths protects the server and still lets the readers through.

When the columns change

A dataset's schema is the list of its columns, with the name, the type and the meaning of each. Rows change at every release, and that is expected. The schema should change rarely, and when it does the change is either compatible or breaking, and each kind has its own rule.

A compatible change adds a column, or widens what a column can hold without changing what it means. A program that reads the file by column name keeps working. Make the change at the current URL, add the column at the end, raise the minor part of the schema version (1.2 to 1.3) and note it in the change log.

A breaking change renames a column, removes one, changes its type, its unit, its code list or the population it counts, or changes what a column means while keeping its name. That last one does harm because nothing fails and the numbers are quietly wrong. Every program that reads the file is affected, so a breaking change gets a new major version at a new path. Publish the new series under /data/senior-enrolments/v2/, keep the old series where it is, and say on the page which old column maps to which new one. If the old series can be produced for one more release, do that, so readers have a release to compare.

A few habits avoid most breaking changes. Keep column names plain, with ASCII letters, digits and underscores, and keep units and years out of them. Put a value that varies into a row: a column per year, such as 2024 and 2025, is a schema change every year, where a year column is a row change. Never put text in a numeric column. A suppressed count is a blank cell with a second column that says it was suppressed, or a documented token such as <5 that is used the same way in every release.

The schema version is a number on the dataset page and in schema.json, separate from the release date. Two releases a year apart with the same columns share a schema version, and a reader can tell from the number alone whether the file they wrote code for still reads.

Describing the columns

Publish the column list as a file a program can read as well as a table a person can. The format this site uses and recommends is Table Schema, a short JSON document naming each column with its type, a one-line description, the list of codes it holds where that applies, and the column that identifies a row. A workbook can carry the same information on a sheet of its own, one row per column, and some Australian publishers already do this well.

Identify a row by a code that does not change, and publish the lookup from code to name beside the data. Schools, councils, stations and hospitals are renamed and merged, and a name that was unique in one release is ambiguous in the next. A dataset keyed on a stable code can be joined to its own earlier releases and to other datasets. The name belongs in the file as well, for the reader, and the code is the key.

The dataset page

One page per dataset, at a URL that does not change, holds the description, the licence with the attribution wording the agency wants, a contact for corrections, the known caveats such as a preliminary period or a change of method, the current and dated download links, the schema version and the change log. Give the page schema.org Dataset markup, and give the portal record the same facts pointing at the current URL, so catalogues and search engines find it. On a CKAN portal, update the existing resource at each release instead of adding a new one, so that the resource id and its URL hold.

A check list

Every release has a dated URL that never changes and is never removed.
A current URL redirects to, or holds a copy of, the newest release, and the file name is the same at both.
versions.json lists every release with its date, URL, hash, row count and schema version.
Files are served with a content type, Last-Modified, ETag and range support, and need no login or challenge.
Column names are plain and stable, values that vary are rows, and numeric columns hold only numbers.
schema.json describes every column, and a schema version separate from the release date says when the columns changed.
A compatible change adds a column at the end. A breaking change starts a new major version at a new path and keeps the old one.
Rows carry a stable code, and the lookup from code to name is published.
The dataset page states the licence, the attribution and a contact, and carries the change log.

What this site does with it

When a publisher follows this layout, this site reads the current URL each week, and a release it has not seen becomes a dated version here with no person involved, with a diff against the version before and the publisher's own file beside it. A column the register does not name is reported and held until a person reviews it, so a breaking change at the source pauses the mirror until it is understood. The publishers page says what else a publisher gets. National Digital, which runs this site, will talk through a layout with any agency that asks on its contact page. No government agency has endorsed this site.