The US EPA listed R134a as unacceptable refrigerant for newly light-duty vehicles manufactured or sold in the United States as of model year 2021. Carbon dioxide CO2 (R744) has been revived as a natural environmentally friendly refrigerant and is considered a strong alternative to R134a as it has a global warming potential (GWP) of 1 compared to 1300 for R134a. In an air-conditioning system and due to the different thermodynamic properties of CO2, the heat rejection process at the high-pressure side will take place above the critical point for high ambient/sink temperatures. Therefore, for a given ambient temperature, the GC pressure (high-side pressure) can be optimized and controlled independently. Either through simulations or experiments, researchers have been focusing on developing control correlations for the GC pressure to maximize the COP using offline control correlations or online methods. Maximizing COP does not mean that the system is working at its highest cooling/heating capacity that might be desired for example in a transient start-up operation to cool down or heat up the car cabin in the shortest possible time. In addition, offline control correlations suffer deviation from the true optimum as they rely on the system model. Online methods, on the other hand, can be more accurate but often lack the fast convergence to the optimum solution. The aim of this thesis was to develop a new strategy to optimize and control the CO2 transcritical air conditioning system for not only optimum COP, but also optimum cooling/heating capacity or a tradeoff solution based on the system state i.e. transient, steady state, or capacity demand. To find the Pareto Front or the best non-dominated solutions between the COP and the cooling capacity for any set of operating conditions, the existing Non-Dominated Sorting Genetic Algorithm II (NSGA-II) is used, and the results are generated based on a transcritical CO2 thermodynamic model. The best solutions of both objectives COP and cooling capacity are presented by a Pareto Front for a given operating conditions. Each solution of the Pareto Front has a unique GC pressure and superheat. An optimization parameter k that ranges from 0 to 1 is introduced to easily select maximum COP, maximum cooling capacity, or any of trade-off solutions. Based on the system operating conditions, the high-level optimizer signals the system actuators, the GC pressure, and superheat reference values. The proposed optimization and control approach can be employed as a hybrid offline and online strategy. Based on the current operating conditions, the high-level optimizer will provide an initial estimate of the optimum solution to the online optimizer, which will start searching for the true optimum online from this close initial guess. An optional online optimizer can be integrated in the loop e.g. before the controller, resulting in conjunction with the offline optimizer in a hybrid solution. Such hybrid solution can reduce the time to approach the desired operating point compared to online only methods. Compared to offline only methods, this can additionally enhance COP and Qc based on the actual system characteristics, while it is also able to adapt to changing system characteristics. While the results in this thesis are presented in terms of the cooling capacity, the same findings can be applied for the heating capacity. For further experimental investigations of the transcritical cycle, a modular transcritical CO2 heat pump system and its coolant system have been constructed at the MSU Turbomachinery Lab that support cooling, heating, and dehumidification modes. Several parameters' effects on the system performance have been analyzed and the experimental results are reported.