The thermodynamic formalism allows one to access the chaotic properties of equilibrium and out-of-equilibrium systems, by deriving those from a dynamical partition function. The definition that has been given for this partition function within the framework of discrete time Markov chains was not suitable for continuous time Markov dynamics. Here we propose another interpretation of the definition that allows us to apply the thermodynamic formalism to continuous time. We also generalize the formalism --a dynamical Gibbs ensemble construction-- to a whole family of observables and their associated large deviation functions. This allows us to make the connection between the thermodynamic formalism and the observable involved in the much-studied fluctuation theorem. We illustrate our approach on various physical systems: random walks, exclusion processes, an Ising model and the contact process. In the latter cases, we identify a signature of the occurrence of dynamical phase transitions. We show that this signature can already be unravelled using the simplest dynamical ensemble one could define, based on the number of configuration changes a system has undergone over an asymptotically large time window.