Scraping German Government Data: What Breaks and Why

What breaks when you scrape government data

It breaks in five places, and the page layout is only one of them. The other four concern content: which version you read, whether the value was corrected afterwards, whether an ordinance exists for it at all, and which day the value applies to. A parser that returns the right number today can return a wrong one tomorrow at any of these points, and no test turns red.

We ran into this while checking our own datasets. The weekly check reads every figure again in the statute text, in the Federal Law Gazette (Bundesgesetzblatt) or in the notice in the Federal Gazette (Bundesanzeiger). The errors that check found serve as examples below, with the API responses so you can verify them.

Five failure points and what helps

Consolidated or promulgated text: which one does your scraper read?

Gesetze im Internet shows consolidated versions, and by the portal's own account they are not official. The official text is the promulgation in the Bundesgesetzblatt, which has been issued electronically on the federal promulgation platform since 1 January 2023. For amending acts it contains only the amending instruction, which publishers and online providers work into the base text.

That has two consequences for a scraper. First, it usually reads the right number from the portal but does not know the promulgation it would have to cite. Second, the update time depends on the portal: it points out that consolidating large amendments can take some time and that the wording changes only when the amendment enters into force. If you need a deadline or a contribution rate on a given date, you read the old or the new text depending on the day.

This hits values that apply in several steps. The dataset social insurance contribution rates therefore carries every rate with its legal basis, source, valid-from and valid-to. The promulgation serves as evidence.

Corrections: the same value promulgated twice

Even a promulgated figure can be wrong and corrected later. The 2024 garnishment threshold notice dated 10 May 2024 is in the Federal Law Gazette (BGBl. 2024 I No. 160). The issue of 24 May 2024 (No. 165a) brought a correction: the increment for the first person entitled to maintenance is 561.43 euros, not 560.90 euros. The portal lists both references in its full citation.

A scraper that ran between the two dates and has since looked only for new notices holds the value from the first version. It does not notice, because the page stays unchanged. The difference is 53 cents per month, but the value feeds into every calculation of the garnishable amount.

At Quellenkontor the corrected value is in the dataset, and the legal basis names both references. The response below shows them for one day in July 2024, and the garnishment thresholds dataset page lists every step since 2015.

First promulgation and correction compared

Value, legal basis and validity in one response

$ curl "https://api.quellenkontor.dev/v1/hr/pfaendungsfreigrenzen?datum=2024-07-15" \
    -H "Authorization: Bearer $QK_KEY"

# Excerpt of the response
{
  "erhoehung_erste_person_monat": 561.43,
  "gueltig_ab": "2024-07-01",
  "gueltig_bis": "2025-06-30",
  "rechtsgrundlage": "Pfändungsfreigrenzenbekanntmachung 2024 vom 10. Mai 2024 (BGBl. 2024 I Nr. 160), berichtigt durch BGBl. 2024 I Nr. 165a (Erhöhungsbetrag für die erste Person 561,43 Euro)",
  "quelle": {
    "titel": "Pfändungsfreigrenzenbekanntmachung 2024 vom 10. Mai 2024 (BGBl. 2024 I Nr. 160)",
    "url": "https://www.gesetze-im-internet.de/pf_ndfreigrbek_2024/BJNR0A00A0024.html"
  },
  "stand": "2026-09-23"
}

When there is no ordinance: the insolvency levy in 2021

Some values have no ordinance of their own, and anyone searching for one finds nothing. Under § 360 SGB III the statutory insolvency benefit levy is 0.15 percent. An ordinance under § 361 SGB III may deviate from it, and the Federal Ministry of Labour and Social Affairs has used that power repeatedly. For 2021 there is no such ordinance. The rate of 0.12 percent sits in § 360 SGB III as amended by the Employment Security Act of 3 December 2020, valid until 31 December 2021.

In our own check this was a finding against us: the legal basis first read § 361 SGB III in our dataset. The value was right, the legal basis was not. We corrected it on 23 September 2026. A scraper would have made the same mistake, because today's § 360 states 0.15 percent and its footnote points only to the ordinances for 2022 to 2024.

For 2022 an ordinance applies again: the Insolvency Benefit Levy Ordinance 2022 sets 0.09 percent. The source thus alternates between statute and ordinance: in 2021 the statute applies, in 2022 an ordinance again. A parser with a fixed search pattern breaks in such years.

Which route fits your source?

What official interfaces already solve

Open, machine-readable access exists, and whoever uses it does not need to scrape. Gesetze im Internet also provides the full texts of statutes and ordinances as XML, expressly for automated processing. The data structure is described by a publicly available DTD, and a daily table of contents is available as XML at gii-toc.xml. The portal's robots.txt blocks nothing.

Statistics have their own access. The Federal Statistical Office offers a RESTful/JSON interface for GENESIS-Online, and the user documentation is available as version 5.1 dated 1 June 2026. According to Destatis, GENESIS-Online can be used free of charge under the Data Licence Germany Attribution 2.0. The licensing questions are covered in the article on official works and copyright.

These routes deliver texts and tables, but not a value with a legal basis and validity per effective date. You have to take the step from statute text to a verified number with a valid-from date yourself, and this step is where this article's examples fail.

What every stored government value should carry

• Legal basis with the reference of the promulgation, not only the URL of the page you read it on.
• Valid-from and valid-to as separate fields, so retroactive runs find the value of the right day.
• Retrieval date and date of the last check against the primary source.
• A note when the value was corrected, with the reference of the correction.
• License or legal status of the source, so passing the data on to customers is clear.

When building it yourself still makes sense

• You need exactly one source with a stable, structured format, such as one GENESIS table for one indicator.
• Your team reviews the values professionally anyway, for example in payroll, and can classify deviations at once.
• You need data no dataset provider carries and accept the upkeep knowingly.
• You want the data only once as a snapshot, without following later corrections.

Quiz on the failure points

Valid from, valid to: why the effective date belongs to the number

A number without an effective date is right on one day only. According to the dataset the garnishment thresholds from 1 July 2024 apply until June 30, 2025, after which the 2025 notice replaces them. Anyone who stores only the latest value can no longer recalculate a payroll run for June 2024.

That is why Quellenkontor responses carry valid-from and valid-to in every answer, as the example above shows. How you correct a payroll run with the values of the right day is described in the article on retroactive payroll corrections.

For checking, the dataset has a weekly run that reads every figure against the primary source. How that run works and which corrections it has found so far is on the page how we verify the values.

Checklist for your government data connection