Field-programmable gate arrays (FPGAs) play an important role in the acceleration of computationally expensive algorithms for machine learning, aerospace, and ASIC prototyping. The emergence of FPGAs in the cloud (cloud FPGAs) has accelerated FPGA adoption in various applications due to their low initial cost and the ability to quickly prototype a design. Multi-tenancy, in which multiple users execute circuitry in the same FPGAs simultaneously with logical isolation, reduces cloud FPGA usage cost and increases FPGA utilization. Multi-tenancy introduces new security challenges, such as remote side-channel and fault injection attacks, that cannot be addressed with traditional countermeasures against attacks. In this dissertation, security challenges that arise from the co-location of an adversary with their potential victim on the same FPGA are studied, along with countermeasures for these challenges. On-chip voltage sensors that are realized using FPGA resources are studied first. These sensors can be used to perform remote power side-channel attacks when co-located with a victim hardware design. Time-to-digital converter (TDC) and ring-oscillator (RO) based circuits are evaluated and compared. The insight gained from this study is then used to carry out remote side-channel attacks. The first successful remote side-channel attack against a hardware accelerator running a machine learning algorithm is performed. The input image of a neural network accelerator for digit recognition is obtained using measurements extracted from an on-chip voltage sensor on the FPGA that does not have physical access to the FPGA supply voltage pins. In the third part of this dissertation, countermeasures that improve the security of users in hostile FPGA environments are described. Countermeasures against remote fault injection attacks are evaluated first. In these attacks, remote power-wasting circuits are used to induce timing faults in the victim circuit or crash the FPGA. Two separate countermeasures against fault injection attacks are described. First, partial reconfiguration as a mitigation to wipe out these power-wasting circuits is studied. Findings on using partial reconfiguration against an Intel Stratix V FPGA are shared. These findings show that partial reconfiguration is not fast enough to suppress the power wasters in an Intel Stratix V FPGA before a fault is injected. Second, a circuit and system-level methodology is developed to detect compute failure conditions due to timing faults induced by fault injection attacks. This approach rapidly suppresses incorrect results and regenerates potentially-tainted computations before they propagate, allowing time for an attacker to be suppressed. Finally, the security of the user FPGA bitstream in a hostile environment is addressed. An FPGA bitstream obfuscation method is developed targeting a Xilinx Kintex 7 FPGA that extracts sensitive logic configuration information from the bitstream and encrypts the information using a key generated by a physically-unclonable function (PUF). This obfuscation method provides a second level of defense against bitstream decryption that could also be used to thwart non-invasive probing attacks.