Safety standards in domains like automotive and avionics seek for deterministic execution (lack of jittery behavior) as a stepping stone to build a certification argument on the correct timing behavior of the system. However, the use of artificial-intelligence (AI) software in safety-critical systems carries several built-in and derivative sources of non-determinism that are at odds with safety standard determinism requirements. In this work we analyze the main sources of non-determinism of autonomous driving (AD) software, as highly representative and compelling example of the use of AI software, deep neural networks (DNN) in particular, in critical embedded systems. Paradoxically, DNN-based software in its inference phase—once the NN structure and weights have been fixed—turns out to consist mainly in matrix multiplications, which are inherently quite time deterministic. Our work focuses on sources of variability and non-determinism in AD software, covering algorithmic elements of AD software, low-level software and hardware computing platform, and data-flow constraints among AD modules. As final contribution of our work, which mainly focuses on problem identification, we develop some prospects on the information and metrics needed to better understand and control the unpredictability and non-determinism of AD software.