dima

on software
Posts from blog by tag python:

Under the Hood of Spense.app: The Code.

This article is a translation and adaptation of my article in Russian.

While Spense v0.2 is under development, I want to talk about the internal organization of the application from a technical perspective. This article is mainly for web developers, and it's written in the corresponding language, so if you're reading and don't understand anything, that's okay, you can just skip it.

In a Nutshell

Backend on Django (Python), frontend on Django templates and Bootstrap, with a pinch of JavaScript and some htmx (not anymore).

Why So Boring?

Sounds not very hype, right. But remember that Spense in its current state is not a full-fledged product. It's more of a prototype, in which I often need to change things and test ideas. If the ideas work, I'll throw this code away and write another; if they don't, I'll just keep it for memory.

So, I chose Django not because I love it (actually, I hate it), but because I've gotten used to it over the last year, and it allows me to prototype the application easily and quickly.

Read more
Spense.app python django css js programming

Docker Buildkit: the proper usage of --mount=type=cache

TL;DR The contents of directories mounted with --mount=type=cache are not stored in the docker image, so it makes sense to cache intermediate directories, rather than target ones.

In dockerfile:1.3 there is a feature of mounting file system directories during the build process, that can be used for caching downloaded packages or compilation artifacts.

For example, the uwsgi package must be compiled every time it is installed, and at first glance, build times can be reduced by making the entire Python package directory cacheable:

# syntax=docker/dockerfile:1.3
FROM python:3.10

RUN mkdir /pip-packages

RUN --mount=type=cache,target=/pip-packages \
      pip install --target=/pip-packages uwsgi

> docker build -t pip-cache -f Dockerfile.pip .
# ...
[+] Building 14.6s (7/7) FINISHED

Looks like everything went well, but the target directory is empty:

> docker run -it --rm pip-cache ls -l /pip-packages
total 0

Something is definitely wrong. You can see that during the build uWSGI was compiled and installed. You can even check it adding ls in the build process:

RUN --mount=type=cache,target=/pip-packages \
      pip install --target=/pip-packages uwsgi \
      && ls -1 /pip-packages

> docker build -t pip-cache --progress=plain -f Dockerfile.pip .
<...>
#6 12.48 Successfully installed uwsgi-2.0.20
<...>
#6 12.91 __pycache__
#6 12.91 bin
#6 12.91 uWSGI-2.0.20.dist-info
#6 12.91 uwsgidecorators.py
#6 DONE 13.0s
<...>

Everything is in its place. But the final image is empty again:

> docker run -it --rm pip-cache ls -l /pip-packages
total 0

The thing is, the /pip-packages catalog, that is inside the image, and the catalog, that is in RUN --mount=type=cache,target=<dirname>, are completely different. Let's try to put something inside this directory and track its contents during the build process:

RUN mkdir /pip-packages \
    && touch /pip-packages/foo \
    && ls -1 /pip-packages

RUN --mount=type=cache,target=/pip-packages \
    ls -1 /pip-packages \
    && pip install --target=/pip-packages uwsgi \
    && ls -1 /pip-packages

RUN ls -1 /pip-packages

> docker build -t pip-cache --progress=plain -f Dockerfile.pip-track .
<...>
#5 [stage-0 2/4] RUN mkdir /pip-packages
      && touch /pip-packages/foo
      && ls -1 /pip-packages
#5 sha256:fb542<...>
#5 0.211 foo  👈1️⃣
#5 DONE 0.2s

#6 [stage-0 3/4] RUN --mount=type=cache,target=/pip-packages
      ls -1 /pip-packages
      && pip install --target=/pip-packages uwsgi
      && ls -1 /pip-packages
#6 sha256:10ed6<...>
#6 0.292 __pycache__            👈2️⃣
#6 0.292 bin
#6 0.292 uWSGI-2.0.20.dist-info
#6 0.292 uwsgidecorators.py
#6 2.802 Collecting uwsgi       🤔3️⃣
#6 3.189   Downloading uwsgi-2.0.20.tar.gz (804 kB)
#6 4.400 Building wheels for collected packages: uwsgi
<...>
#6 13.34 __pycache__            👈4️⃣
#6 13.34 bin
#6 13.34 uWSGI-2.0.20.dist-info
#6 13.34 uwsgidecorators.py
#6 DONE 13.4s

#7 [stage-0 4/4] RUN ls -1 /pip-packages
#7 sha256:fb6f4<...>
#7 0.227 foo  👈5️⃣
#7 DONE 0.2s
<...>
  • 1️⃣ file foo created successfully
  • 2️⃣ the directory with the results of the previous docker build was mounted, and there's no foo file
  • 3️⃣ uWSGI is downloaded, compiled and installed again
  • 4️⃣ an updated uWSGI package appeared in the catalog
  • 5️⃣ only the file foo is left in the directory

This means that --mount=type=cache only works in the context of a single RUN instruction, replacing the directory created inside the image with RUN mkdir /pip-packages and then reverting it back. Also, caching turned out to be ineffective because pip reinstalled uWSGI with a full compilation.

In this case, it would be correct to cache not the target directory, but /root/.cache, where pip stores all the artifacts:

RUN --mount=type=cache,target=/root/.cache \
    pip install --target=/pip-packages uwsgi

> docker build -t pip-cache -f Dockerfile.pip-right .
> docker run -it --rm pip-cache ls -1 /pip-packages
__pycache__
bin
uWSGI-2.0.20.dist-info
uwsgidecorators.py

Now everything is in place, the installed packages have not disappeared.

Let's check the effectiveness of caching by adding the requests package:

RUN --mount=type=cache,target=/root/.cache \
    pip install --target=/pip-packages uwsgi requests
                                                👆

> docker build -t pip-cache --progress=plain -f Dockerfile.pip-right .
<...>
#6 6.297 Collecting uwsgi
#6 6.297   Using cached uWSGI-<...>.whl  👈
#6 6.561 Collecting requests
#6 6.980   Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB)
<...>

pip used the pre-built wheel file from /root/.cache and installed a ready-to-use package from it.

All sources are available on GitHub.

docker cache python