Warning : Parental Guidance Requested
Algeria's APS news agency quoted the foreign ministry as saying that Col Gaddafi's wife Safia, daughter Ayesha and sons Muhammad and Hannibal had crossed into Algeria early on Monday.
The report came as rebels in Libya were trying to overcome the pockets of resistance by Gaddafi loyalists.
Col Gaddafi's whereabouts are unknown.
The arrival of Col Gaddafi's wife and three children in Algeria had been reported to the UN and to Libyan rebel authorities, the Algerian foreign ministry said.
It said they had crossed the border between Libya and Algeria at 08:45 local time (07:45 GMT) on Monday.
The BBC's Jon Leyne in the eastern Libyan city of Benghazi said first word of such a move had already come from Libyan rebel headquarters two days ago, and that at the time, Algerian authorities denied that a convoy of six heavily armoured vehicles had crossed the border.
'Warning' Algeria is an obvious refuge for the Gaddafi family as the two countries have a long border and the Algerian government has still not recognised the rebel National Transitional Council (NTC), our correspondent says.
Muhammed and Hannibal are two of the sons with the least involvement in politics.
There is still no reliable word of the whereabouts of the other sons, although some opposition officials have suggested they may be in or close to the Gaddafi stronghold of Sirte.
Rebel fighters have been pushing towards Sirte, Col Gaddafi's birthplace, and earlier took the small town of Nofilia on their way to the city.
The rebels say they think Col Gaddafi himself may still be in the Tripoli area, our correspondent says.
A spokesman for the NTC said it considered Algeria's sheltering of Col Gaddafi's family an act of aggression and would seek their extradition.
"We have promised to provide a just trial to all those criminals," Mahmoud Shamman told Reuters news agency.
"We are warning anybody not to shelter Gaddafi and his sons."