This isn’t an argument against the standard way of doing things, it is an argument to follow the xdg standard, and use xdg environment variables, rather than creating a new unconfigurable directory in $HOME.

Without knowing more, I would expect it is a dataloader issue: your CPUs are bottlenecked trying to get enough data to your GPUs.

You can add more workers to your dataloader in order to paralyze it, though this can lead to weird parallelization bugs sometimes, so if things start acting weird, that might be a reason.