The Transport Ministry has confirmed three people, including two children, have been found dead, while 39 others have been pulled from the water alive, some after desperately clinging to fishing buoys for hours in the huge swell until help arrived.
Search and rescue teams have still not found the stricken vessel, police said, which was ferrying 118 people around Sulawesi island when it sent out a distress signal late on Saturday afternoon.
The whereabouts of the remaining 76 people is also still unclear, with search and rescue teams being hampered by rough conditions and waves up to five metres high as daylight hours dwindle.
When asked whether the boat had sank, South Sulawesi police spokesman Frans Barung said: "Maybe yes, maybe no".
"The boat has not been found, because the weather has not been good," he added, saying the boat was constructed from fibre glass and was therefore difficult to sink.
Authorities lost contact with the ferry late on Saturday afternoon, and it failed to arrive at its destination in Siwa across the Bay of Bone as scheduled.
Officials dismissed initial reports the boat had sunk, instead claiming it was adrift, having lost engine power.
Police, national search and rescue teams and the navy have been scouring the seas for survivors ever since but have been battling harsh elements.
Local search and rescue head Roki Asikin said powerful waves meant it took three hours to evacuate some survivors to shore.
"These waves were very high," he said.
There had been warnings about extreme weather in the area in the days leading up to the accident, with strong winds and rough seas.