Well not completely; your external drive is not used for save-games… According to the 1406-wave FAQ on the beta-forums, this was posted by the Xbox Team:
Q: Can I store game saves on the external storage device?
A: No. Game saves are automatically stored in the cloud and synced to the Xbox One console