Random decrease in fps

I have been experiencing this issue recently and it is starting to really bug me.
I will be playing with a solid 60fps for around 15 minutes until I receive a decrease in fps where the game will hover around 25-35 fps.
To resolve the issue I have to restart my game however it is starting to become a joke.
I run the game at 3840x2160 (2160p) at high settings and get a solid capped 60fps with a RX Vega 64 and Ry-zen 1700 with 16GB of RAM.
I have checked temps and other statistics however they are all pretty normal.
Any advice would be greatly appreciated.