The traffic signal has been widely used in the transport system with a fixed time interval currently. This kind of setting time was determined based on experience for vehicles to generate a waiting time while allowing pedestrians crossing the street. However, this strict setting causes inefficient problems in terms of economic and safety crossing. In this research, we propose a monitoring algorithm to detect, track and check pedestrian crossing the crosswalk by the patterns of behavior. This monitoring system ensures the safety for pedestrian and keeps the traffic flow in efficient. In this algorithm, pedestrians are detected by using HOG feature which is robust to illumination changes in outdoor environment. According to a complex computation, the parallel process with the GPU as well as CPU is adopted for real-time processing. Therefore, pedestrians are tracked by the relationship of hue channel in image sequence according to the predefined pedestrian zone. Finally, the system checks the pedestrians' crossing on the crosswalk by its HOG based behavior patterns. In experiments, the parallel processing by both GPU and CPU was performed so that the result reaches 16 FPS (Frame Per Second). The accuracy of detection and tracking was 93.7% and 91.2%, respectively.