A memristor device is an electrical component whose internal state depends on past current and/or voltage. It is highly sought-after for various applications due to its non-volatile nature, retaining memory even without power. Integrating memristor devices into crossbar arrays offers significant improvements in throughput and power through in-memory analog computing. The implementation of hardware-based perceptron using memristor arrays has gained substantial interest recently. However, previous demonstrations often relied on software or digital processors for critical functionalities like activation functions which led to frequent analog/digital conversions, extensive back-and-forth communication, and more complex hardware, compromising energy efficiency and computing parallelism. In these implementations, memristor arrays are widely used in multiply-accumulating (MAC) operations while performing parallel vector-matrix-multiplication (VMM). However, achieving VMM operations requires unrolling two-dimensional (2D) inputs into one-dimensional (1D) vectors, requiring data preprocessing and additional computing resources and time. This thesis proposes a novel fully hardware-based activation function designed for implementation between different layers of a perceptron. This innovative approach enables the transmission of analog signals to the subsequent layer without unnecessary digital conversion, communication, or processing. Consequently, the system experiences improved power efficiency and throughput by eliminating power-intensive peripheral circuitry and minimizing unnecessary data movements. Compact rectified linear units (ReLUs) and memristor arrays serve as the building blocks of a two-layer perceptron. Experimental results demonstrate a recognition accuracy of 93.63% for the Modified National Institute of Standard and Technology (MNIST) dataset, comparable to its software-based counterpart. Adopting a fully hardware-based neural network minimizes both data shuttling and conversion, significantly enhances computing throughput and power efficiency. Finally, an innovative three-dimensional (3D) memristor array architecture is proposed to perform matrix-matrix pointwise multiplication, directly multiplying 2D inputs with weight matrices without the need for data unrolling. This yields notable improvements in both energy efficiency and computing throughput. The proposed system finds applications in various domains, such as classification and edge detection in a single iteration. The proposed architecture achieves a recognition accuracy of 94.36% in classifying the MNIST dataset using MATLAB and LTspice. Furthermore, the complete system is designed with printed circuit boards to validate its functionality as a proof-of-concept for feature extraction tasks.