The ageing Hubble Space Telescope has detected light from the dim object, which the researchers think would have formed when the universe was just 480 million years old.
This was a period when galaxy formation in the early Universe was going into "overdrive", said a NASA team involved in the discovery.
Galaxies are the vast archipelagoes of stars filling space, such as our own spiral-shaped Milky Way galaxy. But the new galaxy, called UDFj-39546284, is the most distant and ancient one yet spotted by astronomers.
According to Professor Richard Bouwens of Leiden Observatory in the Netherlands: "We`re seeing these galaxies -`star cities`- that are building themselves up over cosmic time."
The team, comprising US and European astronomers, observed rapid growth over a relatively short period of time. Their data showed there was just one galaxy in existence about 500 million years after the Big Bang.
But this rises to 10 galaxies some 150 million years later. The tally has doubled about 100 million years later.
"You start out with these little seeds in the very early Universe which would eventually have formed stars, then star clusters, baby galaxies then eventually these large majestic galaxies that we know today," according to Professor Bouwens.
"It`s very exciting to see this complicated physical process actually take place somewhere that no man has seen before," Professor Bouwens told BBC News.
Dr Olivia Johnson of the Royal Greenwich Observatory at the National Maritime Museum said that quantifying the rapid evolution of the Universe will reveal a greater detail about what was happening in the early cosmos -such as when the first stars and galaxies formed.
"These are big, open questions in astronomy and the fact that we are finally able to look into the primordial universe for the first time is quite exciting," she said.
According to the researchers, the new image from Hubble will enable astronomers to test their current theories of the evolution of the Universe.
Professor Bouwens stressed that the observation had yet to be confirmed but that he and his colleagues were "pretty confident" that they had discovered the oldest galaxy caught on camera to date.
"There are many different sorts of objects that can masquerade or look very much like these distant objects. We`ve done lots of checks and lots of tests and we think that this candidate is OK," he said.
The researchers detailed their finding in the journal Nature.