The ability to observe temperature variations inside the human body may help in detecting the presence of medical anomalies. Abnormal changes in physiological parameters (such as metabolic and blood perfusion rates) cause localized tissue temperature change. If the anatomical information of an observed tissue region is known, then a nominal temperature profile can be created using the nominal physiological parameters. Temperature-varying radiation emitted from the human body can be captured using microwave radiometry and compared to the expected radiation from nominal temperature profiles to detect anomalies. Microwave radiometry is a passive system with the ability to capture radiation from tissue up to several centimeters deep into the body. Our proposed method is to use microwave radiometry in conjunction with another imaging modality (such as ultrasound) that can provide the anatomical information needed to generate nominal profiles and improve detection of temperature anomalies. An inference framework is developed for using the nominal temperature profiles and radiometric weighting functions obtained from electromagnetic simulation software, to detect and estimate the location of temperature anomalies. The effects on inference performance of random and systematic deviations from nominal tissue parameter values in normal tissue are discussed and analyzed.