Spense.app v0.2
Hey everyone! I've finished working on the next version of Spense with a bunch of improvements and as per tradition I'm sharing the most interesting parts.
Accounts and Wallets Page
In the app interface, you can now manage your wallets and view the current balance:
Read more
Under the Hood of Spense.app: The Code.
While Spense v0.2 is under development, I want to talk about the internal organization of the application from a technical perspective. This article is mainly for web developers, and it's written in the corresponding language, so if you're reading and don't understand anything, that's okay, you can just skip it.
In a Nutshell
Backend on Django (Python), frontend on Django templates and Bootstrap, with a pinch of JavaScript and some htmx (not anymore).
Why So Boring?
Sounds not very hype, right. But remember that Spense in its current state is not a full-fledged product. It's more of a prototype, in which I often need to change things and test ideas. If the ideas work, I'll throw this code away and write another; if they don't, I'll just keep it for memory.
So, I chose Django not because I love it (actually, I hate it), but because I've gotten used to it over the last year, and it allows me to prototype the application easily and quickly.
Read moreSpense.app v0.1
I've rolled out version 0.1 of my app for tracking finances and decided to report on the progress. To be honest, it's not an actual mobile app, but a Progressive Web App - essentially a website that you can open on your phone, add to the Home Screen, and it will launch in full screen without browser panels. I don't yet know how to make real apps, but even in its current form, it works quite well.
By the way, there's even an icon already, which I created in ChatGPT/DALL-E:
As you've already guessed, I named it Spense. I came up with this name a long time ago by trimming the word "expense", and I bought the domain back then, which I'm only now starting to use more or less.
Currently, I'm the only user, there's no registration, and I think at least until version 1.0 everything will remain closed, but I'm happy to show you what's going on there now and what has changed.
Read moreI started making a personal finance app. Why?
Since around 2014, I've been keeping track of my finances in Google Spreadsheets. It always went like this: 2-3 times a week, I'd sit down at the computer, gather receipts, go through the transaction history in banking apps, recall expenses from memory, and record them in a spreadsheet.
The spreadsheet had one row for each day, and columns for accounts, wallets, and a couple of calculated fields. There was also a "Notes" field where I would describe in almost free form where the money went.
As you can see, I didn't exactly have a lot of money. Probably because I mostly ate and drank coffee instead of working, but the point is that I didn't want to economize on these daily things at all. Saving on everyday items and cutting back is fundamentally unnatural for me, and I've always tried to avoid it. Therefore, the question "Where does the money go?" didn't concern me at that moment, but more interesting questions did:
- How much money do I have right now?
- How much did I have a month/six months/a year ago? Have I become richer or poorer?
- Can I afford to spend on a vacation/buy a new phone/go to a private clinic right now? Will I go into the red by the next paycheck?
- What should my income be so that with my current spending, I start saving any money at all?
- How soon will I start starving if I lose my job?
Abstractions and Inheritance in C - Elegant Foot-Shooting
Sometimes you want to abstract and generalize something in C code. For example, if you want to print the contents of a structure multiple times, you end up writing printf("%s %d %f\n", foo->bar, foo->baz, foo->boom) everywhere like a fool, and it intuitively seems that there should be a way to do foo->print(foo), and not just with foo, but with any structure.
Let's take an example: there is a guy with a first name and a last name, and there is a bird that has a name and an owner.
typedef struct Person Person;
struct Person {
char *first_name;
char *last_name;
};
typedef struct Bird Bird;
struct Bird {
char *name;
Person *owner;
};
To print information about these animals, a cunning C programmer would simply write two functions:
void Person_Print(Person *p) {
printf("%s %s\n", p->first_name, p->last_name);
}
void Bird_Print(Bird *b) {
printf("%s of %s %s\n", b->name, b->owner->first_name, b->owner->last_name);
}
And they would be right! But what if we have many such structures and our brains are corrupted by OOP?
Read moreGraphQL and Python
This Week in Changelogs: flask, pytest, IPython, etc
pyenv 2.3.13, 2.3.14
Highlights from the changelog:
- added versions 3.10.10, 3.11.2, and 3.12.0a5;
- fixed versions 3.5.10 and 3.6.15 for macOS and modern 64-bit platforms.
This one made me laugh a bit:
That's how programming actually works!
TIL: head -n123 is a part of POSIX, head -123 is a shorthand that can be missing in some operating systems (pull request).
IPython 8.11.0
Highlights from the changelog:
%autoreloadsupports meaningful parameters (%autoreload all,%autoreload off, etc), not only numbers (%autoreload 0,%autoreload 2, etc).
I like the log of the pull request, it illustrates the approach of implementing a feature step-by-step, one frame at a time:
Also, this fragment is quite interesting, print and logger.info need to be used carefully for logging and protected from being overwritten during hot-reload:
p = print
logger = logging.getLogger("autoreload")
l = logger.info
def pl(msg):
p(msg)
l(msg)
Everything you wanted to know about GitHub actions:
flask 2.2.3
Although the changelog is not that big, I like the thing about flask run --debug.
Previously, it was flask --debug run , and it was awkward. The fix itself is quite small, but there's a lot of changes in docs, and also a PyCharm screenshot was changed. Nice and pure!
pytest 7.2.1, 7.2.2
The changelogs contains mostly bug fixes. One of them is about pytest.approx() causing ZeroDivisionError on dicts.
Another one fixes type checkers behaviour for the following code, which I think should be illegal:
with pytest.raises(RuntimeError) if val else contextlib.nullcontext() as excinfo:
(Please, don't write the code like this.)
And they fixed a race condition when creating directories in parallel, using os.makedirs(..., exists_ok=True). Simple, but helpful.
whitenoise 6.4.0
The changelog mentions support for Django 4.2. It was good to know, by the way, that STATICFILES_STORAGE is going to be changed to STORAGES dict (pull request).
django-cors-headers 3.14.0
- added support for Django 4.2,
- switched from
urlparsetourlsplit.
The latter is the most interesting, urlsplit is slightly faster. Also, it's cached, so sometimes you gain a huge performance.
The difference between these functions is that urlparse includes parsing of the "parameters" section of a URL:
scheme://netloc/path;parameters?query#fragment
^ this
Since it's not widely used, in most cases it's safe to switch from urlparse to urlsplit.
This Week in Changelogs: Django and faker
Django 4.1.6, 4.1.7
- 4.1.6: release notes, blog post
- 4.1.7: release notes, blog post
9d7bd5a
An interesting bug of parsing the Accept-Language header. The format of the header value is complex, so there's a bunch of regular expressions and @functools.lru_cache(maxsize=1000) for caching the result. However, you can pass a huge header multiple times, causing DoS, so they added two if statements:
- one that checks if the length is less than
ACCEPT_LANGUAGE_HEADER_MAX_LENGTH - second - for checking the comma-separated strings. So they decided not to just raise an exception or truncate the string by
[:ACCEPT_LANGUAGE_HEADER_MAX_LENGTH], but truncate the value in a safe way, so it can be parsed in a meaningful result. Good job!
26b7a25
There was a bug in generated SQL, caused by that .desc() in the model's Meta.constraints:
constraints = [
UniqueConstraint(
Lower("name").desc(), name="unique_lower_name"
)
]
which resulted in <...> WHERE LOWER("myapp_foo"."name") DESC <...> when checking the uniqueness. Apparently, Django can check the constraints itself, not delegating it to the underlying database.
Although the fix is trivial, the case is not, and it wasn't covered in the initial implementation.
By the way, I like how they use typographic double quotes:
msg = "Constraint “name_lower_uniq_desc” is violated."
a637d0b f3b6a4f Those black updates are annoying, mainly because they make git blame misleading. However, there's a solution I didn't know about:
git blame --ignore-revs-file <file>- ignore commits listed in thefile..git-blame-ignore-revsfile - make GitHub ignore them as well.
590a92e
The bug was caused by the commit which we've already seen. Now you can safely raise ValidationError without the code.
628b33a
One more DoS fix, now it's about number of opened files when you put too many of them in one multipart payload. The fix introduces TooManyFilesSent exception, which results in HTTP 400 (DATA_UPLOAD_MAX_NUMBER_FILES = 100 by default).
I like this fragment:
try:
return self._parse()
except Exception:
if hasattr(self, "_files"):
for _, files in self._files.lists():
for fileobj in files:
fileobj.close()
raise
Beware of freeing your resources, garbage collector can't help you all the time!
faker 16.6.1..17.0.0
Their CHANGELOG is quite descriptive, so I'll just highlight something that I liked.
fakercan generate valid image URLs using specific websites (TIL), and one of them, PlaceIMG, is shutting down, and they removed it from the list. The announcement is included in all the generated images:
- Biased booleans introduced.
- Added
emojiprovider 🎉 🥳 - Added new
es_ARprovider, but for some reason it's not in the reflected in the CHANGELOG. - Black formatting - always beautiful.
In addition, it turned out that GitHub can put those linter errors from the actions right in the code. I don't know yet how to add this, but I definitely want it!
Go: Delayed file system events handling
Suppose you need to do something when some file system event occurs. For example, restart the web server when files change. Quite a common practice in development: recompile the project and restart it on the fly immediately after editing the source files.
This site is written in Go, and I recently decided to add a similar hot-reload for markdown files with articles: make the web server notice a new file in it's data directories and repopulate the internal in-memory-storage without restarting itself. And I wanted to "listen" to the file system, not scan it every few seconds.
Fortunately, there is already a good listener fsnotify, which can observe given directories. (Not recursively though, but I don't have that many directories.)
The README gives a pretty clear example. I wrapped it in the Watcher() function and added my channel there, which I send something to once an event happen. I don't care about the type of events, so I just always send 1. Also, I wrapped the Watcher() function into another Watch() function, which executes the reload function passed to it at every FS change.
Something like this (error handling and some unimportant stuff intentionally left out):
func main() {
dirs := []string[
"/foo",
"/bar",
]
go Watch(dirs, func() {
reloadStuff()
})
}
func Watch(dirs []string, reload func()) {
ch := make(chan int)
go watcher(dirs, ch)
// Execute the reload function on each
// file system event
for range ch {
reload()
}
}
func Watcher(dirs []string, ch chan int) {
watcher, _ := fsnotify.NewWatcher()
defer watcher.Close()
done := make(chan bool)
go func() {
for {
select {
case event, _ := <-watcher.Events:
// Send a notification to the channel
// on any event from the watcher
ch <- 1
case err, _ := <-watcher.Errors:
log.Println("error:", err)
}
}
}()
for _, dir := range dirs {
watcher.Add(dir)
}
<-done
}
I tried to run ran some tests and found that more than one event can be triggered when changing a file (e.g. CHMOD and WRITE sequentially). And if multiple files are changed at once (git checkout, rsync, touch *.*), there'll be even more events, and my hot-reload is triggered on each of them.
In fact, I only need to trigger it once, if a lot of events came in a short period of time. That is, accumulate them, wait half a second, and if nothing else came, do the thing.
To my shame, I couldn't come up with a good solution on my own, but I noticed that CompileDaemon, which I use in development to recompile the source code, works exactly as I want. The solution from there is elegant (as elegant as it can be in Go), and it's about using time.After(): it starts a timer and sends the current time to the return channel after a specified interval.
As a result, the Watch() function has become the following:
func Watch(dirs []string, reload func()) {
ch := make(chan int)
go watcher(dirs, ch)
// A function that returns the channel, which receives
// the current time at the end of the specified time interval.
createThreshold := func() <-chan time.Time {
return time.After(time.Duration(500 * time.Millisecond))
}
// `threshold := createThreshold()` is also acceptable,
// if you want to trigger reload() on the first run.
// I don't need that, so an empty channel is enough.
threshold := make(<-chan time.Time)
for {
select {
case <-ch:
// At each event, we simply recreate the threshold
// so that `case <-threshold` is delayed for another 500ms.
threshold = createThreshold()
case <-threshold:
// If nothing else comes into the `ch` channel within 500ms,
// trigger the reload() and wait for the next burst of events.
reload()
}
}
}
It worked exactly as I wanted it to, and now I can update all the data files through rsync without restarting the web server within 500ms.
I invented PHP.
Docker Buildkit: the proper usage of --mount=type=cache
TL;DR The contents of directories mounted with
--mount=type=cacheare not stored in the docker image, so it makes sense to cache intermediate directories, rather than target ones.
In dockerfile:1.3 there is a feature of mounting file system directories during the build process, that can be used for caching downloaded packages or compilation artifacts.
For example, the uwsgi package must be compiled every time it is installed, and at first glance, build times can be reduced by making the entire Python package directory cacheable:
# syntax=docker/dockerfile:1.3
FROM python:3.10
RUN mkdir /pip-packages
RUN --mount=type=cache,target=/pip-packages \
pip install --target=/pip-packages uwsgi
> docker build -t pip-cache -f Dockerfile.pip .
# ...
[+] Building 14.6s (7/7) FINISHED
Looks like everything went well, but the target directory is empty:
> docker run -it --rm pip-cache ls -l /pip-packages
total 0
Something is definitely wrong. You can see that during the build uWSGI was compiled and installed. You can even check it adding ls in the build process:
RUN --mount=type=cache,target=/pip-packages \
pip install --target=/pip-packages uwsgi \
&& ls -1 /pip-packages
> docker build -t pip-cache --progress=plain -f Dockerfile.pip .
<...>
#6 12.48 Successfully installed uwsgi-2.0.20
<...>
#6 12.91 __pycache__
#6 12.91 bin
#6 12.91 uWSGI-2.0.20.dist-info
#6 12.91 uwsgidecorators.py
#6 DONE 13.0s
<...>
Everything is in its place. But the final image is empty again:
> docker run -it --rm pip-cache ls -l /pip-packages
total 0
The thing is, the /pip-packages catalog, that is inside the image, and the catalog, that is in RUN --mount=type=cache,target=<dirname>, are completely different. Let's try to put something inside this directory and track its contents during the build process:
RUN mkdir /pip-packages \
&& touch /pip-packages/foo \
&& ls -1 /pip-packages
RUN --mount=type=cache,target=/pip-packages \
ls -1 /pip-packages \
&& pip install --target=/pip-packages uwsgi \
&& ls -1 /pip-packages
RUN ls -1 /pip-packages
> docker build -t pip-cache --progress=plain -f Dockerfile.pip-track .
<...>
#5 [stage-0 2/4] RUN mkdir /pip-packages
&& touch /pip-packages/foo
&& ls -1 /pip-packages
#5 sha256:fb542<...>
#5 0.211 foo 👈1️⃣
#5 DONE 0.2s
#6 [stage-0 3/4] RUN --mount=type=cache,target=/pip-packages
ls -1 /pip-packages
&& pip install --target=/pip-packages uwsgi
&& ls -1 /pip-packages
#6 sha256:10ed6<...>
#6 0.292 __pycache__ 👈2️⃣
#6 0.292 bin
#6 0.292 uWSGI-2.0.20.dist-info
#6 0.292 uwsgidecorators.py
#6 2.802 Collecting uwsgi 🤔3️⃣
#6 3.189 Downloading uwsgi-2.0.20.tar.gz (804 kB)
#6 4.400 Building wheels for collected packages: uwsgi
<...>
#6 13.34 __pycache__ 👈4️⃣
#6 13.34 bin
#6 13.34 uWSGI-2.0.20.dist-info
#6 13.34 uwsgidecorators.py
#6 DONE 13.4s
#7 [stage-0 4/4] RUN ls -1 /pip-packages
#7 sha256:fb6f4<...>
#7 0.227 foo 👈5️⃣
#7 DONE 0.2s
<...>
- 1️⃣ file
foocreated successfully - 2️⃣ the directory with the results of the previous
docker buildwas mounted, and there's nofoofile - 3️⃣ uWSGI is downloaded, compiled and installed again
- 4️⃣ an updated uWSGI package appeared in the catalog
- 5️⃣ only the file
foois left in the directory
This means that --mount=type=cache only works in the context of a single RUN instruction, replacing the directory created inside the image with RUN mkdir /pip-packages and then reverting it back. Also, caching turned out to be ineffective because pip reinstalled uWSGI with a full compilation.
In this case, it would be correct to cache not the target directory, but /root/.cache, where pip stores all the artifacts:
RUN --mount=type=cache,target=/root/.cache \
pip install --target=/pip-packages uwsgi
> docker build -t pip-cache -f Dockerfile.pip-right .
> docker run -it --rm pip-cache ls -1 /pip-packages
__pycache__
bin
uWSGI-2.0.20.dist-info
uwsgidecorators.py
Now everything is in place, the installed packages have not disappeared.
Let's check the effectiveness of caching by adding the requests package:
RUN --mount=type=cache,target=/root/.cache \
pip install --target=/pip-packages uwsgi requests
👆
> docker build -t pip-cache --progress=plain -f Dockerfile.pip-right .
<...>
#6 6.297 Collecting uwsgi
#6 6.297 Using cached uWSGI-<...>.whl 👈
#6 6.561 Collecting requests
#6 6.980 Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB)
<...>
pip used the pre-built wheel file from /root/.cache and installed a ready-to-use package from it.
All sources are available on GitHub.
Fast commit and push
I want to share a shell-function called gacp (Git Add, Commit and Push), which I came up with a few months ago and have been using about every hour since then:
# fish
function gacp
git add .
git commit -m "$argv"
git push origin HEAD
end
# bash/zsh
function gacp() {
git add .
git commit -m "$*"
git push origin HEAD
}
Usage example:
> gacp add some new crazy stuff
[master fb8dcc9] add some new crazy stuff
<...>
Enumerating objects: 12, done.
<...>
To github.com:foo/bar.git
912c95d..fb9dcc9 master -> master
No more chains with && and quotes for verbose messages!