Homes without heat would become uninhabitable as temperatures fell, state Governor Andrew Cuomo said.
New York City Mayor Michael Bloomberg put the figure at 30,000-40,000 people.
At least 106 US deaths - 40 of them in New York City - have been blamed on Sandy, which struck on 29 October.
Residents who had so far refused to leave their homes would have no other option, Mr Cuomo told a news conference on Sunday.
He also said there would be increasing pressure on public transport on Monday, as more people returned to work and the schools re-opened.
Fuel shortages were easing, but Mr Cuomo urged New Yorkers not to hoard petrol, saying more supplies were on their way.
New York City opened warming shelters in areas without power and handed out blankets to residents who insisted on staying in homes without power.
Temperatures fell to 39F (4C) on Sunday and are forecast to go as low as 30F (-1C) on Monday.
About 730,000 people in New York state still do not have electricity, including more than 130,000 in New York City, the governor said.
Nearly a million people in the neighbouring state of New Jersey remain without power, and petrol is being rationed.
Hundreds of runners who had been planning to take part in the New York marathon - cancelled by Mayor Bloomberg on Friday - joined impromptu runs to raise funds or deliver aid.