The recordings did not make clear why the pilot left the cockpit or why he could not regain entry as the plane steadily descended toward a mountain range in a remote area of the French Alpine region on Tuesday.
Investigators were studying the voice recordings from one of the "black boxes" for answers on Thursday while the search continued for a second black box.
"The guy outside is knocking lightly on the door and there is no answer," an unnamed investigator told the Times, citing the recordings.
"And then he hits the door stronger and no answer. There is never an answer."
"You can hear he is trying to smash the door down," the investigator added.
A spokesman for Lufthansa, whose budget arm Germanwings operated the flight, said the carrier was aware of the Times story, adding: "We have no information from the authorities that confirms this report and we are seeking more information. We will not take part in speculation on the causes of the crash."
The retrieval of one of two cockpit recordings came as French President Francois Hollande, Germany's Angela Merkel and Spain's Mariano Rajoy travelled to the crash site to pay tribute to the 150 victims, mostly Germany and Spanish.
France's BEA air incident investigation bureau was not immediately available for comment on the Times report but earlier had said it was too early to draw meaningful conclusions on why the plane went down.
"We have not yet been able to study and to establish an exact timing for all the sounds and words heard on this file."
Jouty expected the first basic analysis in "a matter of days" but warned this read-out could be subject to errors and that more work would be needed for a full interpretation.
Although he said "words" had been heard on the tape, Jouty would not confirm whether that meant the Airbus A320's pilots were conscious and he gave no details of the recordings.
The BEA said the plane started descending a minute after reaching cruising height and lost altitude for over nine minutes. The last words of the pilot to the ground confirmed the next navigational waypoint, ending with a call-sign and "thank you".
The New York Times quoted a senior military official involved in the investigation as saying the cockpit audio showed "very smooth, very cool" conversation between the pilots in the early part of the flight. The audio then indicated one of the pilots left the cockpit.
"We don't know yet the reason why one of the guys went out," the official added. "But what is sure is that at the very end of the flight, the other pilot is alone and does not open the door."
The Lufthansa spokesman said that since the September 11, 2001, hijacked plane attacks on the United States, cockpit doors cannot be opened from the outside, in line with regulations.
Interior Minister Bernard Cazeneuve earlier said a terror attack was not the main hypothesis. Jouty said, however, that no theory could be excluded yet.
While stressing it was too early to form a clear picture, he ruled out a mid-air explosion and said the crash scenario did not appear to be linked to depressurisation.
Jouty said the second, data black box had not been found and said he could not confirm Hollande's remark that its casing had been discovered.
Among the new details, he said the airliner had flown in a straight line directly into the mountain, but would not say whether that seemed to be at the hand of a pilot or auto-pilot.
Germanwings said 72 Germans were killed in Tuesday's crash, the first major air passenger disaster on French soil since the 2000 Concorde accident just outside Paris. Spanish officials said 51 Spaniards were among the victims.
As well as Germans and Spaniards, victims included three Americans, a Moroccan and citizens of Britain, Argentina, Australia, Belgium, Colombia, Denmark, Israel, Japan, Mexico, Iran and the Netherlands, officials said. However DNA checks to identify them could take weeks, the French government said.
Police and forensic teams on foot and in helicopters investigated the crash site about 100 km (65 miles) north of Nice.
"When we go to a crash site we expect to find part of the fuselage. But here we see nothing at all," pilot Xavier Roy, coordinating air operations, said of the confetti of debris.
Roy said teams of investigators had been dropped by helicopter onto the site and were working roped together at altitudes of around 2,000 meters (6,000 feet).
Germanwings said it had cancelled one flight on Wednesday and was using 11 planes from other carriers for about 40 flights after some of its crew members had refused to fly.