A federal judge on Tuesday rejected a settlement deal Google hammered out with publishers over its controversial Google Books archive, saying the proposed agreement went too far in giving Google control over the digitalization of books.

“The question presented is whether the 1/8 settlement agreement 3/8 is fair, adequate, and reasonable,” Judge Denny Chin of the U.S. District Court in New York City wrote in his 48-page ruling. “I conclude that it is not.”

The decision is the latest twist in a saga that is helping to shape the legal and commercial landscape around digital publishing.

Google’s stated mission is “to organize the world’s information and make it universally accessible and useful.” As part of that goal, it struck a deal in 2004 with several major libraries to digitally copy the books in their collections. The company now has an archive of more than 12 million publications.

To those who want information to be online and accessible, that’s a great idea. To publishers — and many of the authors they work with — it’s a terrifying one.

Their representatives quickly slapped Google with a lawsuit, which later gained class-action status. After two years of negotiations, Google hacked out a settlement deal with two industry groups, the Authors Guild and the Association of American Publishers. A modified version of that deal got a preliminary seal of approval from the court in late 2009.

But the deal triggered hundreds of objections, and Judge Chin took a step back to consider them before giving the deal his final approval. His ruling on Tuesday makes clear that the intensity of the backlash — and the specific arguments made by critics of the deal — prompted him to change his mind.

“Many of the concerns raised in the objections would be ameliorated if the 1/8 agreement 3/8 were converted from an ‘opt-out’ settlement to an ‘opt-in’ settlement,” he wrote.

Google’s settlement agreement is a complex, 166-page document. While the company took pains to protect the rights of copyright holders — only tiny snippets are revealed from in-print books — it put the burden on authors and publishers to police their works’ inclusion in the archive. Google will remove books on request, but without an explicit request, it will otherwise digitize anything it can get hold of.

That didn’t sit well with Judge Chin. He also expressed concern over the agreement’s handling of “orphaned” books — works that are still under copyright, but no longer in print.

“The questions of who should be entrusted with guardianship over orphan books, under what terms, and with what safeguards are matters more appropriately decided by Congress than through an agreement among private, self-interested parties,” Chin wrote in his ruling.

Google said it is considering its next step in a legal battle that has already dragged on for six years.

“This is clearly disappointing, but we’ll review the court’s decision and consider our options,” a company spokeswoman said. “Like many others, we believe this agreement has the potential to open-up access to millions of books that are currently hard to find in the U.S. today.”

Judge Chin acknowledged the appeal of the “utopian dream” of a vast digital archive, but warned against letting Google be its overseer.

“While the digitization of books and the creation of a universal library would benefit many, the 1/8 agreement 3/8 would simply go to far,” he wrote. “The 1/8 deal 3/8 would give Google a significant advantage over competitors, rewarding it for engaging in wholesale copying of copyrighted works without permission.”