Same problem here, posted about it in this thread: PSA: Possible fix for long loading times - Forza Horizon 4 / FH4 Discussion - Official Forza Community Forums
Really hoping for a fix before EOL. Especially before the final season starting end of July.