Block sanding body filler could be broken up into 3 different stages; roughing in with 40 grit, straightening it with 80 grit, and finalizing it for primer with 150 grit.
40 grit works the best when it’s used while the filler is still tacky, and not yet dry. The question is, when to stop using 40 grit and switch to 80 grit.
40 grit is only to ‘rough in’ the shape of the filler. The filler does not need to be ‘straight’ before you switch to 80 grit. The amount of time using 40 grit depends on how much filler is being used; if an entire panel has a heavier spread on it, it will probably require more time 40 grit to get a closer shape. You don’t want to create a ton of extra work for 80 grit.
Stop with the 40 grit before hitting metal. This is assuming a whole area is being skim coated. It might take some practice to find the right balance of when to switch. But, a lot of the block sanding and getting the filler straight should be done with 80 grit. This leaves plenty of material to get ready for primer. And, you will be able to spread additional layers of filler over 80 grit, which is far better than over 40 grit.
When the filler is either straight, or ready for another coat, there should be no 40 grit scratches remaining. If there is, you probably used 40 grit for too long.