As Facebook becomes Meta, data it obtained from Lithuanian company could be key to the metaverse

Facebook was sued more than two years ago over allegations it illegally copied and shared the world’s largest repository of digitized and realistic three-dimensional objects and scenes. It was a case that, at the time, didn’t seem to square with the main mission of the world’s biggest social network.

The planned jump by the company now known as Meta Platforms into the metaverse, however, now makes clear the possible stakes in the US litigation, and the potential value of the 3D database owned by a once-obscure Lithuanian home-design startup called Planner 5D.

Meta’s research scientists are racing to let computers “see” the same way people see — to achieve the seemingly simple but technologically formidable task of recognizing that the cylindrical paper object resting on the flat surface is a coffee cup on a table. Planner 5D’s database of more than a million digitized objects and scenes offer data that, the company says in that suit against Meta, is critical to training computer vision algorithms.

At a court hearing today, Meta asked a federal judge in San Francisco to tighten the secrecy in the case, in which the former Facebook and Princeton University are accused of scraping, copying and sharing Planner 5G’s unique trove of 3D objects and scenes.

That kind of data can train an artificial intelligence system to convert a string of pixels and numbers into the recognition: “This is a coffee cup.” Or, as a video Meta recently released as part of its public posts about its ambitious plans to help build the metaverse, to teach computers to distinguish between a Boston terrier, a border collie and a Shetland sheepdog.

“The goal of computer vision is to try and develop techniques that allow you to basically go from that set of numbers to understanding what is in that image,” Laurens Van Der Maaten, research director at Facebook AI Research, said on the Meta video linked to the company’s metaverse videos.

Computer vision, Chief Executive Mark Zuckerberg said in a conversation during an Oct. 28 keynote address at the company’s annual developer conference, is one area of technology where Meta needs a breakthrough to realize its plans for the metaverse.

“Over time, I hope we’re seen as a metaverse company,” Zuckerberg said during his address with the chief scientist for Facebook Reality Labs, Michael Abrash. “The metaverse is the next frontier, just like social networking was when we got started.”

Corporate secrets

Meta is concerned that the Planner 5D case could expose sensitive corporate secrets, while Planner 5D says the restrictions Meta is proposing are unprecedented and unnecessary.

The former Facebook, saying it is “one of the most scrutinized and reported on companies in the world,” with its internal documents “the target of misuse and leaks, resulting in substantial reputational harm,” asked US District Judge William Orrick to take the unusual step of blocking Planner 5D from allowing those internal documents to be printed or downloaded, and in some cases even transported outside the US, even behind the protective shield of the lawyers in the case.

Meta was burned, it said, in litigation against Six4Three, a software developer that built apps for the Facebook platform, when documents sealed in that case by a California court were transported to the UK by Six4Three’s founder and leaked to a member of Parliament, causing major regulatory scrutiny for Facebook. Meta continues to litigate that case.

Planner 5D countered that Meta’s proposed restriction “would dramatically impair collaboration” with its lawyers. “No court that Planner 5D is aware of has ordered such a severe encumbrance on an unwilling party,” the Vilnius, Lithuania-based company told Orrick.

Neither Meta nor US-based lawyers for Planner 5G responded to questions from MLex this week.

Planner 5G says it spent years and millions of dollars developing what has become a unique resource for training computer vision AI. Princeton scraped the entirety of it, and it then shared more than 45,000 files with Facebook and others, including digital depictions of everything from structural home features like windows, to pieces of furniture, to outdoor objects such as trees, paths, lawns, barbeques and swimming pools.

“Planner 5D owns a collection of over a million hand-crafted, digitized, and realistic three-dimensional objects and scenes, depicting a wide variety of household and office designs. To Planner 5D’s knowledge, no other collection in the world numbers even in the tens of thousands,” the company said in a 2020 complaint against Facebook and Princeton that alleged copyright violations.

Planner 5D was founded a decade ago as a web-based home-design tool that would allow people to design their homes or offices with a computer. The data embedded in those images and renderings allowed users to toggle between two- and three-dimensional presentations of the design, and to rotate them in space to get a better view, Planner 5D said.

Over time, the digital renderings of such mundane items as sofas and beds have become critical for computer scientists developing computer vision, Planner 5D says, and it has pivoted its business to invest “significantly in developing algorithms that capitalize on Planner 5D’s catalog of three-dimensional files to achieve market-leading 3D recognition.”

Even before Facebook changed its name to Meta and said its future lies in the metaverse, Planner 5D said in court filings that the market for AI-enhanced software is expected to grow to $60 billion by 2025.

First-ever ruling

Orrick in April denied motions by then-Facebook and Princeton to dismiss Planner 5G’s allegations that the companies violated copyright law by "scraping" its renderings to develop computer-vision technology, allowing the case to move forward.

"This case has been stalled in the pleadings stage for almost two years; there is no reason to wait any longer,” the judge wrote in the order. The case is scheduled to go to trial in March 2023.

At a video hearing today, Orrick asked whether he would be the first US judge to rule on a dispute to limit the availability of electronic documents to a party based outside the US when the party didn’t agree with the limitation. “I think you would be the first judge that would actually assess the pros and cons of it,” answered Dale Cendali, a lawyer for Meta.

Cendali said Internet connections between the US and Lithuania are good enough for Planner 5D’s personnel to view documents without downloading or printing them, and that Meta would run the risk of a breach by hackers if the documents were downloaded to computers outside the US.

But lawyers for Planner 5D said Meta’s proposal would block lawyers in the case from traveling anywhere outside the US with case documents on their laptops, even if the laptops were encrypted. And in terms of security, “frankly, that’s a red herring. Putting a computer in the United States does not make it any less hackable,” said Christian Andreu-von Euw, a lawyer for Planner 5G.

Orrick said that while he understood Meta’s concerns given the Six4Three leak in the UK, he's “inclined” to reject the social media giant’s proposal, because Planner 5D’s needs to easily access documents outweigh Meta’s concerns.

“I understand the concern,” Orrick told Cendali. “There’s some substance behind it. But I’m concerned, as I balance the equities here, you’re dealing with a party with a lot less resources, that needs not only to know what’s going on, but to actually help in the technical implementation of the case.”