Designing wind farms in complex terrain is an important task, especially for countries with a large portion of complex terrain territory. To tackle this task, an optimization framework is developed in this study, which combines the solution from a wind resource assessment tool, an engineering wake model adapted for complex terrain, and an advanced wind farm layout optimization algorithm. Various realistic constraints are modelled and considered, such as the inclusive and exclusive boundaries, minimal distances between turbines, and specific requirements on wind resource and terrain conditions. The default objective function in this framework is the total net annual energy production (AEP) of the wind farm, and the Random Search algorithm is employed to solve the optimization problem. A new algorithm called Heuristic Fill is also developed in this study to find good initial layouts for optimizing wind farms in complex terrain. The ability of the framework is demonstrated in a case study based on a real wind farm with 25 turbines in complex terrain. Results show that the framework can find a better design, with 2.70% higher net AEP than the original design, while keeping the occupied area and minimal distance between turbines at the same level. Comparison with two popular algorithms (Particle Swarm Optimization and Genetic Algorithm) also shows the superiority of the Random Search algorithm.