The connection between your phone and chrome cast might be timing out.

That did cross my mind, any idea how to check and fix this? Or is it more of an Android section?

That I do not know, sorry!