dima

on software
Posts from blog by tag python:

Parsing numbers from JSON in Python

Consider this hypothetical endpoint in a Python-based web-server:

# withdraw.py
import json
from flask import Flask, request

app = Flask(__name__)

STATE = {"balance": 1000}

@app.post("/withdraw")
def withdraw():
    """
    Withdraw a requested amount from the user's balance
    """
    try:
        data = json.loads(request.data)
    except ValueError:
        return "Malformed JSON", 400
    if not isinstance(data, dict):
        return "JSON object expected", 400

    # Amount validation
    amount = data.get("amount")
    if not isinstance(amount, (int, float)):
        return f"Invalid type: {amount.__class__}", 400
    if amount <= 0:
        return f"{amount} is too small", 400
    if amount > STATE["balance"]:
        return f"{amount} is greater than your balance", 400

    STATE["balance"] -= amount

    return f"Success! New balance: {STATE['balance']}", 200

Here we want to focus on the logic of the JSON input validation, assuming that the STATE dictionary is used primarily for simplification. Our intention in this part of the code is to:

  • support the amount field as numeric;
  • allow withdrawal only for positive amount;
  • reject a withdrawal if the amount exceeds the balance.

All these conditions are seemingly covered by those three if statements. Let's test it out by running the flask dev server and sending some payloads:

$ flask --app withdraw:app run
 * Serving Flask app 'withdraw:app'
<...>
 * Running on http://127.0.0.1:5000
Press CTRL+C to quit
$ curl 0:5000/withdraw --json '{"amount": 100}'
Success! New balance: 900
$ curl 0:5000/withdraw --json '{"amount": 0}'
0 is too small
$ curl 0:5000/withdraw --json '{"amount": 9999999}'
9999999 is greater than your balance
$ curl 0:5000/withdraw --json '{"amount": 900}'
Success! New balance: 0
$ curl 0:5000/withdraw --json '{"amount": 900}'
900 is greater than your balance

Basic sanity check is passed.

NaN

What if we pass one of the non-standard JSON constants, like NaN?

$ curl 0:5000/withdraw --json '{"amount": NaN}'
Success! New balance: nan

Oops! Apparently, native json parser in Python also accepts this constant, converting it to float("nan") and allowing arithmetic operations.

Examples
In [1]: 1000 - float("nan")
Out[1]: nan

In [2]: 1000 + float("nan")
Out[2]: nan

In [3]: float("nan") * 0
Out[3]: nan

In [4]: float("nan") - float("nan")
Out[4]: nan

So once NaN is passed, it corrupts the state by replacing it with float("nan") and allowing an unlimited number of withdrawals:

$ curl 0:5000/withdraw --json '{"amount": NaN}'
Success! New balance: nan
$ curl 0:5000/withdraw --json '{"amount": 100}'
Success! New balance: nan
$ curl 0:5000/withdraw --json '{"amount": 1000}'
Success! New balance: nan

If we try to persist it in a database though, things get even more interesting: SQLite converts it to NULL, while Postgres can store NaN as an actual numeric value, corrupting the balance in a different way.

SQLite example
# withdraw_sqlite.py
import json
import sqlite3

from flask import Flask, request

app = Flask(__name__)

DB = "app.db"

with sqlite3.connect(DB) as conn:
    conn.executescript("""
        CREATE TABLE IF NOT EXISTS account (
            id INTEGER PRIMARY KEY,
            balance REAL
        );
        INSERT OR IGNORE INTO account (id, balance)
        VALUES (1, 1000);
    """)

@app.post("/withdraw")
def withdraw():
    try:
        data = json.loads(request.data)
    except ValueError:
        return "Malformed JSON", 400
    if not isinstance(data, dict):
        return "JSON object expected", 400
    amount = data.get("amount")
    if not isinstance(amount, (int, float)):
        return f"Invalid type: {amount.__class__}", 400
    if amount <= 0:
        return f"{amount} is too small", 400

    with sqlite3.connect(DB) as conn:
        row = conn.execute(
            "SELECT balance FROM account WHERE id = 1"
        ).fetchone()

        balance = row[0]
        if amount > balance:
            return f"{amount} is greater than your balance", 400

        balance -= amount

        conn.execute(
            "UPDATE account SET balance = ? WHERE id = 1",
            (balance,),
        )

    return f"Success! New balance: {balance}", 200

Let's start the server, send a couple of requests and check the database state:

$ flask --app withdraw_sqlite:app run
<...>
$ sqlite3 app.db "SELECT * FROM account"
1|1000.0
$ curl 0:5000/withdraw --json '{"amount": 123}'
Success! New balance: 877.0
$ sqlite3 app.db "SELECT * FROM account"
1|877.0
$ curl 0:5000/withdraw --json '{"amount": NaN}'
Success! New balance: nan
$ sqlite3 app.db "SELECT * FROM account"
1|
$ sqlite3 -cmd ".nullvalue NULL" app.db "SELECT * FROM account"
1|NULL

So this NaN was actually converted to NULL value. From now on the balance is broken and we can't operate on it anymore. When we try to withdraw a valid amount:

$ curl 0:5000/withdraw --json '{"amount": 123}'

the request fails with HTTP 500, and the webserver raises TypeError:

[2026-08-16 17:22:29,328] ERROR in app: Exception on /withdraw [POST]
<...>
  File "<...>/withdraw_sqlite.py", line 37, in withdraw
    if amount > balance:
       ^^^^^^^^^^^^^^^^
TypeError: '>' not supported between instances of 'int' and 'NoneType'
127.0.0.1 - - [16/Aug/2026 17:22:29] "POST /withdraw HTTP/1.1" 500 -

Of course, adding a NOT NULL constraint will successfully mitigate the problem of corrupted data:

CREATE TABLE IF NOT EXISTS account (
    id INTEGER PRIMARY KEY,
    balance REAL NOT NULL
);
$ curl 0:5000/withdraw --json '{"amount": NaN}'
<!doctype html>
<html lang=en>
<title>500 Internal Server Error</title>
<...>
  File "<...>/withdraw_sqlite.py", line 42, in withdraw
    conn.execute(
    ~~~~~~~~~~~~^
        "UPDATE account SET balance = ? WHERE id = 1",
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        (balance,),
        ^^^^^^^^^^^
    )
    ^
sqlite3.IntegrityError: NOT NULL constraint failed: account.balance
127.0.0.1 - - [16/Aug/2026 17:26:38] "POST /withdraw HTTP/1.1" 500 -

However, moving validation to the persistence layer and returning HTTP 500 is generally a bad idea.


Luckily, json.loads() accepts a parse_constant handler, which is called once NaN, Infinity or -Infinity is encountered. And it's perfectly fine to raise a ValueError from that handler:

# ...

def constant_parser(v):
    raise ValueError(
        f"JSON constant {v} is not supported"
    )

@app.post("/withdraw")
def withdraw():
    try:
        data = json.loads(
            request.data, parse_constant=constant_parser
        )
    except ValueError as e:
        return f"Invalid type: {e}", 400
    # ...
$ curl 0:5000/withdraw --json '{"amount": NaN}'
Invalid type: JSON constant NaN is not supported
$ curl 0:5000/withdraw --json '{"amount": Infinity}'
Invalid type: JSON constant Infinity is not supported
$ curl 0:5000/withdraw --json '{"amount": 123}'
Success! New balance: 877

Problem solved! We don't corrupt the data anymore, and don't allow to withdraw more money than you own.

Implicitly converted types: Booleans and Floats

There are two edge cases that are not covered yet: booleans and large numbers implicitly converted to float("inf").

$ curl 0:5000/withdraw --json '{"amount": true}'
Success! New balance: 999
$ curl 0:5000/withdraw --json '{"amount": false}'
False is too small
$ curl 0:5000/withdraw --json '{"amount": 1e309}'
inf is greater than your balance
  1. Booleans. We've been taught since forever that types in Python should be checked not by using comparison to type(x), but by using isinstance(x, type[s]). Well, in Python bool is a subclass of int, and if we want to harden our API contract, one simple way is to use an exact type check, type(amount) not in (int, float).

  2. 1e309 == float("inf"). The E-notation in Python represents a float, and Python's JSON decoder parses such numbers as floats as well. The largest possible 64-bit double-precision floating-point number is ~1.79×10308. A value with a magnitude greater than that is converted to infinity during the JSON parsing phase, and it should be additionally checked with the math.isfinite() function.

Those cases are not catastrophic, and some of them are already caught during validation against the balance, and, by the way, treating true as 1 can be considered correct by some C programmers.

However, that kinda violates the contract, since we explicitly should accept only numeric values, rejecting all other types at the earliest possible stage, so let's adjust the validation:

import math

# ...

@app.post("/withdraw")
def withdraw():
    # ...
    amount = data.get("amount")
    if type(amount) not in (int, float):
        return f"Invalid amount type: {amount.__class__}", 400
    if not math.isfinite(amount):
        return "Non-finite numbers are not supported", 400
    # ...

Now it looks like all the invalid cases are solved:

$ curl 0:5000/withdraw --json '{"amount": 1e999}'
Non-finite numbers are not supported
$ curl 0:5000/withdraw --json '{"amount": -1e999}'
Non-finite numbers are not supported
$ curl 0:5000/withdraw --json '{"amount": Infinity}'
Invalid type: JSON constant Infinity is not supported
$ curl 0:5000/withdraw --json '{"amount": NaN}'
Invalid type: JSON constant NaN is not supported
$ curl 0:5000/withdraw --json '{"amount": true}'
Invalid amount type: bool
$ curl 0:5000/withdraw --json '{"amount": false}'
Invalid amount type: bool

Although... How will it handle a really large integer number now?

$ curl 0:5000/withdraw --json \
    '{"amount":99<300 more 9s>9999999}'
<!doctype html>
<html lang=en>
<title>500 Internal Server Error</title>
<...>
[2026-08-16 23:12:18,997] ERROR in app: Exception on /withdraw [POST]
<...>
    if not math.isfinite(amount):
           ~~~~~~~~~~~~~^^^^^^^^
OverflowError: int too large to convert to float
127.0.0.1 - - [16/Aug/2026 23:12:18] "POST /withdraw HTTP/1.1" 500 -

Oh dear. So if the isfinite() function encounters an integer, it tries to convert it to float first, and fails. In other words:

  • 1e309 becomes an inf,
  • float(10**309) raises OverflowError,
  • int('9' * 400) is a valid Python integer.

Meaning that we don't need to check finiteness of integers at all:

if type(amount) is float and not math.isfinite(amount):
    return "Non-finite numbers are not supported", 400
$ curl 0:5000/withdraw --json \
    '{"amount":99<300 more 9s>9999999}'
99<300 more 9s>9999999 is greater than your balance

$ curl 0:5000/withdraw --json \
    '{"amount":99<300 more 9s>9999999.0}'  # NB .0
Non-finite numbers are not supported

Okay, I guess that's it, the validation looks really promising, and we managed to cover all cases. The code's become ugly though: quite a lot of frictions only to validate that the numbers are actually numbers.

Pydantic model

That's why we have Pydantic, which can handle this complexity almost for free with a pinch of additional annotations:

from typing import Annotated
from flask import Flask, request
from pydantic import BaseModel, Field, FiniteFloat, ValidationError

app = Flask(__name__)

STATE = {"balance": 1000}

class WithdrawRequest(BaseModel):
    amount: Annotated[
        # rejects NaN, +/-Infinity and overflowing
        # floats such as 1e309
        FiniteFloat,
        Field(
            # rejects true/false and stringified "123.45"
            strict=True,

            # replaces `if amount <= 0:`
            gt=0,
        )
    ] | Annotated[
        # preserves integers as Python ints instead
        # of forcing them through FiniteFloat
        int, Field(strict=True, gt=0)
    ]

@app.post("/withdraw")
def withdraw():
    try:
        data = WithdrawRequest.model_validate_json(request.data)
    except ValidationError as e:
        error = e.errors()[0]
        return error["msg"], 400

    amount = data.amount
    if amount > STATE["balance"]:
        return f"{amount} is greater than your balance", 400

    STATE["balance"] -= amount

    return f"Success! New balance: {STATE['balance']}", 200

Let's double check what we have now:

$ curl 0:5000/withdraw --json '{"amount": NaN}'
Input should be a finite number
$ curl 0:5000/withdraw --json '{"amount": 1e999}'
Input should be a finite number
$ curl 0:5000/withdraw --json '{"amount": Infinity}'
Input should be a finite number
$ curl 0:5000/withdraw --json '{"amount": true}'
Input should be a valid number
$ curl 0:5000/withdraw --json '{"amount": false}'
Input should be a valid number
$ curl 0:5000/withdraw --json '{"amount": "123"}'
Input should be a valid number
$ curl 0:5000/withdraw --json '{"amount": 99<...>999}'
99<...>999 is greater than your balance
$ curl 0:5000/withdraw --json '{"amount": 99<...>999.0}'
Input should be a finite number

Epilog

What a relief, everything is correct, just as expected. Finally: git add ., git commit -m... Oh wait:

$ curl 0:5000/withdraw --json '{"amount": 2.34}'
Success! New balance: 997.66
$ curl 0:5000/withdraw --json '{"amount": 2.34}'
Success! New balance: 995.3199999999999
$ curl 0:5000/withdraw --json '{"amount": 2.34}'
Success! New balance: 992.9799999999999

Sigh... # TODO: Replace with decimals, commit, push, see ya!

Building a Toy Database: Learning by Doing

Ever wondered how databases work under the hood? I decided to find out by building one from scratch. Meet Bazoola - a simple file-based database written in pure Python.

Why Build a Toy Database?

As a developer, I use relational databases every day, but I never truly understood what happens when I INSERT or SELECT. Building a database from scratch taught me more about data structures, file I/O, and system design than any tutorial ever could.

Plus, it's fun to implement something that seems like magic!

Read more

The Hidden Cost of Test Inheritance

I'm subscribed to Adam Johnson's blog and usually really enjoy his writing - it's practical, deep, and no-bullshit. But one recent post, Python: sharing common tests in unittest, caught me off guard.

It describes a "neat" pattern: write reusable test logic in a base class, subclass it to test multiple objects, hiding the base class from unittest discovery. While the intent is fine - DRYing out duplicated test code - the result is fragile, confusing, and just not worth it.

Here's why.

Read more

Under the Hood of Spense.app: The Code.

This article is a translation and adaptation of my article in Russian.

While Spense v0.2 is under development, I want to talk about the internal organization of the application from a technical perspective. This article is mainly for web developers, and it's written in the corresponding language, so if you're reading and don't understand anything, that's okay, you can just skip it.

In a Nutshell

Backend on Django (Python), frontend on Django templates and Bootstrap, with a pinch of JavaScript and some htmx (not anymore).

Why So Boring?

Sounds not very hype, right. But remember that Spense in its current state is not a full-fledged product. It's more of a prototype, in which I often need to change things and test ideas. If the ideas work, I'll throw this code away and write another; if they don't, I'll just keep it for memory.

So, I chose Django not because I love it (actually, I hate it), but because I've gotten used to it over the last year, and it allows me to prototype the application easily and quickly.

Read more

Docker Buildkit: the proper usage of --mount=type=cache

TL;DR The contents of directories mounted with --mount=type=cache are not stored in the docker image, so it makes sense to cache intermediate directories, rather than target ones.

In dockerfile:1.3 there is a feature of mounting file system directories during the build process, that can be used for caching downloaded packages or compilation artifacts.

For example, the uwsgi package must be compiled every time it is installed, and at first glance, build times can be reduced by making the entire Python package directory cacheable:

Read more