High Performance Computing (HPC) has delivered tremendous improvements in scientific applications these days, much of which can be attributed to the development of multiprocessor systems. Non-Uniform Memory Access (NUMA) is widely used today in multiprocessor systems because it allows the execution of massive simultaneous tasks using a large number of cores and high memory Bandwidth (BW). However, adding more processors may not necessarily improve performance. Taking advantage of this architecture demands careful consideration of potential performance pitfalls, which include programming limitations, such as poor scheduling, parallelization and synchronization overhead, or hardware limitations, such as memory and memory BW. Thus, efficient parallel programming and effective data distribution among the cores are the primary steps to achieve high performance for parallel applications. Performance analysis could help users to detect programming and architectural limitations and to gain more insight into HPC applications, thus optimizing performance. Performance analysis investigates a parallel application and determines targets for optimization. This optimization could lead to better execution time or less memory BW usage. In this research, we focus both on programming and hardware limitations in parallel applications. We first discuss programming limitations and different factors that affect an application's performance. We provide an extensive study of language features and runtime scheduling systems of commonly used threading parallel programming models for HPC, including OpenMP, Intel Cilk Plus, Intel TBB, OpenACC, Nvidia CUDA, OpenCL, C++11 and PThreads. We also evaluate the performance of OpenMP, Cilk Plus and C++11 for data and task parallelism patterns on CPU using a set of benchmarks. We show that performance varies with respect to factors such as runtime scheduling strategies, parallelism and synchronization overhead, load balancing and uniformity of task workload among threads. Such assessment provides a guideline for users to choose a proper API and best parallelism pattern for their applications. In addition, we show the impact of memory BW as a hardware limitation on HPC applications. We provide a quantitative study of high bandwidth memory (HBM) for a set of memory and computation intensive HPC applications. We indicate that HBM improves the performance of both memory and computation intensive applications. However, the improvement of computationally intensive applications is less in comparison to memory intensive applications. The importance of memory BW in NUMA architecture and its great influence on HPC application performance encouraged us to introduce a top-down method for memory BW prediction for HPC applications. Using only a few data points and application abstractions, we estimate memory bandwidth usage for unknown problem sizes and other processor numbers in NUMA with both statistical methods and supervised machine learning algorithm. This research also provides valuable insights on BW tend for different HPC applications (regular and irregular).