A new kind of microscope that ties together footage from dozens of smaller cameras can provide researchers with 3D views of their experiments, whether it's recording the behavior of hundreds of freely swimming zebrafish or the grooming behavior of fruit flies across a vast field of view.
Stitching together video from dozens of cameras allows for a unique 3D view of macroscopic experiments in microscopic detail.
When two smart graduate students took the first look with their pieced-together microscope, it turned out better than they had hoped. Sure, there was a hole in one section, but the others were still able to find Waldo.
The duo resolved their software problems and demonstrated a successful proof-of-principle device on the classic children's puzzle book. By combining 24 smartphone cameras into a single platform and stitching their images together, they created a single camera capable of taking gigapixel pictures over a space approximately the size of a paper.
Researchers made an unprecedented discovery after six years, several design iterations, and one startup company.
"It's like human vision," said Roarke Horstmeyer, a Duke University assistant professor of biomedical engineering who is also an assistant professor of biological engineering. "We were stunned when our students studied zebrafish using it for the first time."
A new kind of microscope that stitches together video from dozens of smaller cameras can provide researchers with 3D photographs of their experiments, whether recording 3D video of the behavior of dozens of freely swimming zebrafish or the grooming behavior of fruit flies across a broad field of view.
Horstmeyer and his colleagues demonstrate the versatility of their new high-speed, 3D, gigapixel microscope, which was previously used in close collaboration with Dr. Eva Naumann's lab at Duke University. The new version of MCAM takes 3D measurements, provides better detail at smaller scales, and produces better films.
The MCAM's highly parallelized design poses its own data processing challenges, as a few minutes' worth of recording can yield thousands of hours of data. "Our algorithms combine physics with machine learning to fuse the video streams from all the cameras and recover 3D behavioral information across space and time," said Kevin C. Zhou, a postdoctoral researcher in Horstmeyer's lab.
Matthew McCarroll of the University of California – San Francisco is studying the behavior of zebrafish exposed to neuroactive drugs. Researchers may uncover new therapies or better understand existing ones by looking for variations in behavior.
Thanks to the MCAM's all-encompassing view, McCarroll and his group describe unexpected movements they'd never seen before. They were able to see variations in the fish's pitch, whether they sat at the top or bottom of their tanks, and how they observed prey.
"We've long been constructing our own systems with single lenses and cameras, which have worked excellent for our purposes," says McCarroll, an independent scientist pursuing pharmaceutical chemistry in the UC system's professional researcher series. "It's amazing to see what a legit physicist can come up with to improve our experiments."
Michel Bagnat, a Duke professor of cell biology, works with zebrafish. However, the researchers instead focus on how the animals evolve from an egg to a fully formed adult on a cellular level.
With the help of the new MCAM, researchers have demonstrated that they can achieve all of these measurements while the fish live their lives unencumbered.
Jennifer Bagwell, a researcher and lab manager at the Bagnat lab, believes that this microscope will dramatically alter how a lot of developmental biologists do their research. "Especially if it turns out that anesthesia is a problem," she said.
Horstmeyer envisions that this work would enable for larger automated parallel experiments, such as watching a plate with 384 wells filled with various organoids to monitor potential therapeutic reactions, and then autonomously indicating any findings of interest.
"The modern laboratory is becoming more automated every day, with large well plates now being filled and maintained without ever touching a human hand," Horstmeyer said. "This is creating demands for new technologies that can help streamline the tracking and recording of data."
Horstmeyer has launched a startup company called Ramona Optics, founded by coauthor Mark Harfouche, who created the first photograph of Waldo. One of its early licensers, MIRA Imaging, is using the technology to "fingerprint" fine art, collectibles, and luxury goods to inoculate against forgery and fraud.
Additional photos of the microscope in action may be found here:
- MCAM project webpage: https://mcam.deepimaging.io/
- Sample videos: https://gigazoom.rc.duke.edu/
- Open source code: https://github.com/kevinczhou/3D-RAPID
- Sample raw video files: https://doi.org/10.7924/r4db86b1q
Kevin C. Zhou, Mark Harfouche, Colin L. Cooke, Jaehee Park, Pavan C. Konda, Thomas Doman, Paul Reamey, Veton Saliu, Clare B. Cook, Maxwell Zheng, John P. Bechtel, Aurélien Bègue, Matthew McCarroll, Jennifer Bagwell, Gregor Horstmeyer, Michel Bagnat, and Roarke Horstmeyer, 20 March 2023, Nature Photonic
The Office of Research Infrastructure Programs (ORIP), the Office of the Director, the National Institutes of Health's National Institutes of Health (R44OD024879), the National Cancer Institute (NCI) of the National Institutes of Health (R43EB030979), the National Science Foundation (2036439), and a Duke Coulter Translational Partnership Award.
