The space of phylogenetic mixtures for equivariant models

Abstract

The selection of the most suitable evolutionary model to analyze the given molecular data is usually left to biologist's choice. In his famous book, J Felsenstein suggested that certain linear equations satisfied by the expected probabilities of patterns observed at the leaves of a phylogenetic tree could be used for model selection. It remained open the question regarding whether these equations were enough for characterizing the evolutionary model. Here we prove that, for equivariant models of evolution, the space of distributions satisfying these linear equations coincides with the space of distributions arising from mixtures of trees on a set of taxa. In other words, we prove that an alignment is produced from a mixture of phylogenetic trees under an equivariant evolutionary model if and only if its distribution of column patterns satisfies the linear equations mentioned above. Moreover, for each equivariant model and for any number of taxa, we provide a set of linearly independent equations defining this space of phylogenetic mixtures. This is a powerful tool that has already been successfully used in model selection. We also use the results obtained to study identifiability issues for phylogenetic mixtures.