The nature of the “Hakka Yimin Worship,” one of the Taiwan folk beliefs, has caused controversy in the academic circles in Taiwan Province. The dominant Hoklo scholars claim that the Hakka Yimin Worship is of a “ghost” nature, which offend the Hakka scholars and the Yimin Temple. The cause could be traced back to a historical hidden scar between the two dialect groups hundreds of years ago in the social development of the Qing Dynasty(1636-1912). Back then the narratives of the “Hakka Yimin Worship” by the gentry of the two groups were already different, which was then has to do with the “armed confrontations between the Hakka and the Hoklo in Taiwan.” According to the Hakka Yimin belief, the Yimin Temple in Fangliao (枋寮) area in Hsinpu (新埔) Township, Hsinchu (新竹) County and the one in Miaoli (苗栗) County were built at the same time. They are also the oldest in northern Taiwan. However, the good attendance of the former temple leads to more historical materials and research achievements related. Therefore, this paper explores the historical narratives of the temple for over two hundred years in Hakka’s own words and in others’ words to investigate the relationship between the Hakka and Hoklo ethnic groups and the interaction between their relationship and the Hakka folk belief in Taiwan Province in the Qing Dynasty.