Modern semiconductor manufacturing requires photolithographic printing of subillumination wavelength features in photoresist via electromagnetic energy scattered by complicated photomask designs. This results in aerial images which are subject to constructive and destructive wave interference, as well as electromagnetic resonances in the photomask features. This thesis proposes a 3-D full-wave frequency domain nonconformal Octree mesh based Finite Element Method (OFEM) electromagnetic scattering solver in combination with Fourier Optics to accurately simulate the entire projection photolithography system, from illumination source to final image intensity in the photoresist layer. A rapid 1-irregular octree based geometry model mesher is developed and shown to perform remarkably well compared to a tetrahedral mesher. A special set of nonconformal 1st and 2nd order hierarchal OFEM basis functions is presented, and 1st order numerical results show good performance compared to tetrahedral FEM. Optical and modern photomask phenomenology is examined, including optical proximity correction (OPC) with thick PEC metal layer, and chromeless phase inversion (PI) masks.