Climate change, security of supply and local air pollution are among the challenges that are shaping the future of energy systems worldwide. In response to these challenges, various goals are set nationally and internationally that energy systems are supposed to fulfil. These include e.g. EU 20-20- 20 targets or the 100% renewable energy system in Denmark in 2050. Nevertheless, the ultimate result with regard to the respective future energy systems remains an open question.This cumulative PhD thesis deals with application and development of energy system optimisation models to address the various challenges stemming from energy use. It consists of a generic part and eleven papers focusing on different, but related topics. The generic part of the thesis serves to provide the background for the papers as well as to place them within the literature and to highlight various linkages between them. The challenges of climate change, security of supply, and local air pollution are addressed in the papers by focusing on renewable energy systems, demand side management options, climate change mitigation and resource potentials. In the process of the study the energy system optimisation models, Balmorel and TIAM, were further developed in order to enhance their capabilities.Methodologically, this thesis heavily relies on mathematical programming and scenario analysis. This is mainly a consequence of the tools that are used (both models are formulated as linear optimisation problems) and the nature of the issues that are dealt with (i.e. high degree of uncertainly with regard to future technology characteristics, global policy development on climate mitigation, etc.). Additionally, geographic information systems are used in one of the papers to conduct a spatial analysis for estimating wind energy potentials.One of the most interesting results obtained in the course of this study is related to a methodology for estimating wind energy potentials developed in one of the papers. It is shown that the economic wind energy potential based on microscale wind climate data can be twice as high compared to the results obtained with simulated mesoscale data (i.e. representing usually used type of data). This has a number of interesting implications for energy system modelling studies on both regional and global level; namely, more realistic wind potential estimates, higher competitiveness of wind energy, and an increased climate change mitigation potential.Other results highlight among others, the possible future roles of individual technologies (i.e. wind power in Denmark and carbon capture and storage in China) in the climate constrained world, the difficulty to achieve the 2°C target agreed upon in the Copenhagen Accord and the importance of large emerging economies in this context, and the role of district heating in the future Danish renewable energy system.The outcomes of this work illustrate the potential of energy system optimisation models to address challenges of the future. The model development that was undertaken can be used by energy planners, modellers and researches in order to improve reliability of their analysis. Moreover, the results of this study can be useful to policy-makers in order to inform their policy decisions with regard to climate change mitigation and energy system development.