I play on both PC and Xbox One and on both platforms I have come to experience the controller (Microsoft Xbox One Controller Minecraft Green on PC, connected via Bluetooth; Microsoft Xbox One Controller Sea of Thieves on Xbox One X) randomly just stop working.
On PC it will pretty regularly and seemingly only when in actual races just do one action (like it will start turning hard left) and not accept any input anymore. Turning it off will turn off the wrong behavior, but it won’t allow me to continue the race after turning it back on. Instead I have to complete the race with the keyboard, after being returned to free roam the controller will be responsive again. The only button that stays active through this is the guide button.
On console I noticed something similar, but different. While I always retain my basic controls, randomly the Y button for rewinds will stop working. Happened so far for me only in street scene races.
Both controllers are fully charged. I replaced the batteries to be sure it wasn’t just a side effect of low power. I don’t have any other controller issues with the same controllers on the same platforms in dozens of hours of Sea of Thieves, Forza Horizon 3 and Forza Motorsport 7.