In recent years, research on user's activity recognition using a smart phone has attracted a lot of attentions. A smart phone has various sensors, such as camera, GPS, accelerometer, audio, etc. In addition, smart phones are carried by many people throughout the day. Therefore, we can collect log data from smart phone sensors. The log data can be used to analyze user activities. This paper proposes an approach to inferring a user's physical activities based on the tri-axis accelerometer of smart phone. We propose recognition method for four activity which is physical activity; sitting, standing, walking, running. We have to convert accelerometer raw data so that we can extract features to categorize activities. This paper introduces a recognition method that is able to high detection accuracy for physical activity modes. Using the method, we developed an application system to recognize the user's physical activity mode in real-time. As a result, we obtained accuracy of over 80%.