Pictures of Dayashankar Singh visiting Baidyanath Temple, one of the twelve Jyotirlingas, surfaced on Wednesday.
According to sources, Singh visited the temple on Saturday with three other men.
Singh on Tuesday moved the Allahabad High Court bench here seeking a stay on his possible arrest and challenging the FIR lodged against him by BSP leaders for using "derogatory" language against BSP supremo Mayawati.
An FIR was lodged against Singh on July 20 at Hazratganj police station here under the Scheduled Castes and the Scheduled Tribes (Prevention of Atrocities) Act for passing "derogatory" comments against Mayawati.
The complaint, which was lodged by national secretary of BSP Mewalal Gautam, alleged that Singh's remarks had hurt the feelings of BSP workers and the Dalit community across the country and were aimed at provoking them.
Referring to Singh's comments telecast on national TV channels, he had said they were so derogatory that they could not be mentioned in the complaint and so a CD of the speech was being submitted.
"Singh was aware that his offensive comments could trigger a strong reaction and riot," the complaint said, adding this amounted to an offence under the SC/ST Act and a case was needed to be lodged against him.
Singh, who was state BJP vice president, was expelled by the party last week for six years in the wake of the storm triggered by his remarks against the BSP supremo.
(With inputs from PTI)