High-speed signal processing is essential for real-time displays in medical imaging applications. Photoacoustic tomography provides structural, functional, and molecular imaging with high resolution in a noninvasive way. Especially, three-dimensional image reconstruction, functional imaging, and real-time display require fast signal processing. Here, we provide a high-speed signal processing method using a graphic processing unit (GPU) to reconstruct ultrasound or photoacoustic B-mode images for real-time displays. The signal processing speed was improved by parallel processing of the beam formation and the envelop detection required for image reconstruction using a massive number of GPU cores. The time using a GPU was 2.778 ms, on average, to process a single-frame B-mode image with $128{\times}3200$ pixels while it was about 3.165 seconds using a central processing unit (CPU). The processing time using a GPU was short enough to reconstruct three-dimensional images for real-time displays.