The 29-year-old, from Langeloop village near the Swazi and Mozambican border, was not asked to plead to a charge of rape when he appeared, a Sapa correspondent reported.
Magistrate Sibonisiwe Sibyi postponed the matter to Monday for a bail application.
The father was arrested at his house on Monday.
Tonga police spokesman Lieutenant Mzwandile Nyambi said the boy was raped on June 29, allegedly while visiting his father during the school holidays.
Nyambi, who spoke outside court, said the boy and his eight-year-old brother lived with their mother at Sikhwahlane village.
“They went to visit their father for school holidays. Their mother learnt about the abuse of the four-year-old after the eight-year-old disclosed the matter to their aunt,” said Nyambi.
He said the boy was taken to hospital where it was confirmed he had been raped.