The social potential of Urban Air Mobility (UAM) as a greener and faster transportation system in and around urban environments is indisputable. Nevertheless, the success of UAM introduction and its wide use will strongly depend on acceptance by the citizens and future UAM users. The impact on overall quality of life, as a multidimensional concept that encompasses physical health, mental and emotional well-being, economic status, education, and the environment, is becoming a significant issue. This paper aims to describe the performance framework for the assessment of the social and environmental impact of UAM. The specific objectives are to identify the full range of UAM’s impacts on citizens’ quality of life and to propose a set of indicators that enables the quantification and assessment of the identified impacts. Firstly, the main issues (focus areas) were identified, namely, noise, visual pollution, and privacy concerns, followed by access and equity, economic aspect, emissions, public safety, and impact on wildlife. In the next step, for each identified focus area, performance indicators were defined along with the several cross-cutting areas for a geographical, temporal, demographic, socioeconomic, and behavioral resolution. The proposed performance framework could enable more efficient mitigation measures and possibly contribute to wider adoption of the UAM operations.