When Will It Be Done - Part 9
THE WORK TODAY, BUILDING THE UNIVERSE
So since early 2014, the primary and often sole task on the film has been the opening section. Although other volunteers have continued work on improving and finishing other sections, that work won’t be wrapped up until the opening section is ready to go. The rest of film is still about 90% complete but most of that work has been stalled as I’ve had to do a huge amount of unexpected work on the opening section myself.
To go from the Big Bang to the surface of the Earth using only photographs, you need a huge amount of photographic data. The first question obviously, Is there even data to pull this off at all? That was the question years ago, back in 2009–2010, when the final vision of the film came together. The answer, “no, but yes.”
No, because the entire universe has not been photographed; in fact, only the tiniest of fractions has been photographed at any resolution. And unlike Cassini, only in one direction — from Earth looking out to the edge of space. And as you go farther away, further back in time, the images are blurrier, until it’s all noise.
But yes, because as a filmmaker I know you only need to see what the “camera” sees. Just like a movie shot on a sound stage. You don’t need to build sets for every room, hallway, and hangar in the Death Star, just the ones the camera will see. All the Death Star sets in all the Star Wars films and media combined probably are .01% of the Death Star and I’m being generous.
So I researched what was out there and that formed our plan. We figured out that we had enough to pull off the shot with photographs of objects placed to scale in accurate positions. I also knew that the shot had to be “going backwards,” as our point of view in photographs is always looking from Earth outwards.
OPENING SECTION SOURCES
So where exactly do we get photographs to cover over 13 billion years of time and space?
Big Bang. This was always a stumper, as the early universe is just elemental particles — no photographs exist. But during the last five years quantum microscopes have imaged atoms and electrons, including the hydrogen atom, the building block of the universe. For a long time, I was going to take the artistic license and recreate this first using atoms, then using galaxy data and the cold inflation model. Some beautiful animations resulted (see below) but as it would be the only section of film not strictly photographic, I decided sound only until we could use real photographs only. Based on this “Sounds of the Big Bang“.
Right after the Big Bang. We have used image data from the Cosmic Background Explorer, removing the false color.
There are no photographs of the first stars, so we move too fast to see these.
Hubble and ESO sources have been used for the earliest galaxies, from 13 billion years ago.
The Sloane Digital Sky Survey (SDSS) is used for background and some foreground planes for the rest of the journey until we reach our Milky Way, and then as background for the Milky Way. Hubble, the European Southern Observatory (ESO), and others are used for high-resolution foregrounds of galaxies and nebulae.
Many sources are used for the journey from edge of the solar system to surface of Earth.
THE TEAMS
There are actually three teams working on the opening section. The Hubble/ESO high-resolution foreground object team led by Jason Harwell started work in mid-2012 and ended up with a five-person team that finished work in mid-2015. It includes an astronomer, Dr. Steve Danford, the head of the physics department at the University of North Carolina, Greensboro (in my home town), who supervised making things accurate, and is an official science advisor to the film.
The current SDSS team is led by Bill Eberly. Six others, including me, have contributed to that team. Another couple of physicists work on that team especially to help with very complex astrophysics mathematics involved.
The final team is the animation team. Well, team in this case means me. I always hoped to get animation help, but it turns out this type of animation is very hard, and there is no reason anyone else would want to learn it, since it can’t be used anywhere else. So I’m the sole animator on the film.
THE PROCESS
The process began by choosing our flight path through the universe, which is also a journey through time. We mapped out where and when in the universe all of the high-resolution sources we had available were located (allowing only those we could process in a “reasonable” amount of time). We then compared this to the areas of the Universe for which there was enough SDSS data to fill the backgrounds. It was actually a very complicated process to figure and map out in such a way that we could visualize it.
Once we had a flightpath, Jason’s High Resolution Team processed the planes of the destination galaxies and nebula along the flight path. They worked with Dr. Danford to be sure that their layers worked as accurately as possible, given our current scientific knowledge.
Bill’s SDSS team downloaded 350,000 SDSS “fields,” which are long, rectangular strips of the sky photographed by telescopes with the corresponding database data of the names, locations, and distances of the galaxies in each field — a few hundred to a few thousand each. There were several million potential galaxies in there.
So our process, in other sections, of manually cutting out the galaxies from the black of space was not an option here. Since no tools existed, we built our own. Bill’s team created code that would automatically cut out galaxies and tag each of them with the relevant location and distance data from the SDSS database. It took months to get this working right.
We then farmed out among ourselves the process of running this code on the 350,000 SDSS fields. It took several weeks, but in the end — by late 2015 — we had approximately 5.5 million individual photographs of galaxies, each tagged with its location in the sky and distance from Earth. A remarkable achievement in itself.
But as Bill and I looked at them, a very obvious and glaring problem emerged. We had several hundred thousand images of other things we didn’t want. Unfortunately, none were real alien UFOs or giant space monsters, which at least would have been newsworthy. They were stars in our own Milky Way, jet and satellite trails, and other mundane IFOs (identified flying objects). This was a bit of a punch in the stomach — we had not realized how much of this was in SDSS data.
Again we discussed the options, and the only practical one was to go through all 5.5 million photographs and remove the bad apples. It was beyond the capability of computing to write a program that could do it perfectly, or even close, and attempting it would take many months anyway. And we would still have to inspect them manually in the end – software is just not smart enough.
We considered farming the job out to volunteers, but after testing I realized that it was impossible for a number of reasons: the judgment required to decide which items to remove, the speed and capacity of the computers needed to handle folders of tens of thousands of galaxies, the need for very large monitors and controlled lighting conditions to be sure consistent decisions were made – it all meant to have any hope of ending up usable within our means, it would have to be a single person working full-time.
The only practical plan was for me stop everything and do it myself. But given how committed we were, I did not agonize over it too much. It had to be done, or give up in failure. I hoped it would take two to three months. It took six months, with two weeks off in the middle. It was the same thing every hour for every day I worked on it:
Load up a folder of SDSS cutouts. Anywhere from one to 100,000 galaxies, plus garbage, would be in the folder. Build a thumbnail view of folder in Adobe Bridge.
Go through all the thumbnails at a size that corresponded to the largest size they would appear on the giant screen.
Remove the local stars, jet trails, etc., without a good galaxy candidate in them.
Then go back and open photographs with a good galaxy candidate, but also containing a local star or jet trail and trim that out to leave the good galaxy candidate behind.
Backup and do the next folder.
I have to be honest. It’s the hardest thing I ever did in my life. It took intense visual cortex concentration to do it. I could only do it eight hours a day (six during the last month), six days a week (five in the last month).
After the first few weeks, I thought I was losing my mind: I was starting to have really strange stuff in my head from staring at tiny objects all day, listening to music. My arms, wrists, and hands were a wreck. But I realized that by doing such an intense right-brained, repetitive activity, my left brain was losing it. I found I could listen to podcasts and audiobooks without an issue; in fact, it helped tremendously. I bought a gaming mouse with programmable macros, used disability features in the operating system, and cut my mouse clicks by over 80%.
Here’s a short timelapse of typical day cleaning including running backups.
I finished intact. Well, sort of. I burst out crying when I finished. And I needed a shot in my right arm for mouse elbow. But by August 2015 we were ready to start assembly.
SDSS TILES
Of course, like any other construction project, there is always a tile job. Our background SDSS planes are massive gigapixel images with millions of galaxies. But animation software cannot handles images that size as a single image. So, as we’ve done in the rest of the film, we build very large images, take them apart to avoid memory issues, and then bring them back together again.
It took us four months to get the assembly process built, tested, and stable enough for the big run. During Thanksgiving week 2015, I fired off Bill’s tile assembly code on ten machines running custom Java code and Photoshop scripts to create the 289 tiles that will make our background universe. Each tile is 10,000 by 10,000 pixels, so it’s a huge amount of data, about 250 gigapixels of image data for this alone.
The process was slow but steady. Only a couple of crashes occurred, due to external USB drives not handling hundreds of thousands of files. The last two tiles finished two months later, January 25, 2016.
Bill and I were enormously satisfied to see how critical my six months of cleaning up was even just for the background tiles. I aimed for a 99.5% accuracy rate; 99.9% would have taken me three or four years of work; 99 percent would have taken me three or four months, but left cleanup that would have caused irreparable damage to the rendered tiles.
The 99.5% accuracy rate means just a few days of cleanup and the tiles are ready to go***.
EDIT!!!!!!!!!!!! UPDATE!!!!!!!!!!!!!!!!!!!!!!
*** this is a perfect example of how blazing a trail for the first time and predicting the future rarely if ever works. That “few days of cleanup” I wrote (this blog serious was written mostly over Christmas holiday 2015) – it was three months!!! However, the tiles are looking gorgeous now – however impossible to post on the web right now due to their size but we will post them later.
NEXT STEPS
Now that we have the tiles, the next step is reassembling those tiles into very-high-resolution background planes, arranged in scientifically accurate locations. Then combine those SDSS cutouts that are close enough to our flightpath that parallax is a factor (our midground in the multi-plane). We have a created a process to place each of them in their scientifically accurate positions using the SDSS science data. Then the foreground is added, the layers of high resolution galaxies from Jason Harwell’s team, and the full multi-plane flythrough of the galaxy are ready to animate and render.
We are working RIGHT NOW on the first and most complex setup, Arp273, right now and footage is forthcoming.
LARGEST MUTIPLANE ANIMATION IN HISTORY
This will result in the largest, longest, most complex multi-plane animation in film history, by a factor of 150,000! Disney used up to seven planes in their most complex shots. In my research I have not heard of anyone doing foreground, midground, and background multi-plane animations to simulate live action using more than seven planes.
And the background tiles is nearly 8.3 Terapixels – the largest composite image processing task by a factor of ten according to Wikipedia.
We will have over 1.2 million planes in our midground, about 10,000 foreground planes, and 12 background planes. The film could be made theoretically with a physical camera, as a computer is not required for multi-plane. But our master background plane for the opening sequence would have to be 0.6 miles (1 kilometer) wide, with a camera system that could go from a few inches to 3 miles (5 kilometers) above it! Not to mention 1.2 million plates of varying sizes of massive, seamless, and perfectly transparent glass.
And I would love (not) to be the person keeping dust, birds, and low-flying aircraft out of the camera rig!