Matomo uses a database for most things, but after I started using the container version of the app I realised that if it saved anything to disk, my upgrades would destroy it because they create a new container and destroy the old one.

So I reached out to the wonderful people at Matomo who said the following:

Matomo doesn't store much on disk. The most important is the "config/config.ini.php" which holds important configuration. In addition, the "misc/" folder can hold custom files like the GeoIP database, custom branding logos, etc. Matomo stores temporary data in "tmp/", but that can be recreated if it gets deleted. And all plugins you install from marketplace get extracted into the "plugins/" directory.

In sum, backup your config file. Also backup the misc and plugins folders if you do not have a setup that will fill that in with your logos and plugins when you create the image.