What can big data teach us about the Big Bang?

Imagine being able to look back in time to the Cosmic Dawn, when the first stars formed in our universe. Or more deeply understand the current era of accelerating celestial expansion. The Square Kilometer Array (SKA) is the world’s largest astronomy project, founded to unravel these and other enigmas that astrophysicists have grappled with for decades.

The SKA is an international effort to build the world’s largest radio telescope – with a collecting area of approximately one square kilometer. Set to begin construction in 2016, The SKA project’s major site will be in South Africa with a second large operation center in Australia, and thousands of antennae located across the world.

Collectively, SKA will be 50,000 times more sensitive than any existing radio instrument and its image resolution quality will dwarf that of the Hubble Space Telescope by 50 times.

Dr. Russ Taylor is the founding director of the Inter-University Institute for Data Intensive Astronomy (IDIA), a newly formed partnership of four South African universities that supports SKA with research conducted by the country’s top scientists.

“SKA will produce tremendous amounts of data, taking big data to the extreme,” he said. “Handling this data, turning it into information that can be shared and distributed easily, and extracting knowledge from it, will be one of the primary challenges for the project. It is a huge opportunity for the South African scientific and technology community.”

SAP Helps The Universe Run Faster

This is where SAP comes in. Mirroring the SKA project, SAP has assembled an international team to figure out if SAP big data technologies can help deal with this cosmic data.

Elke Simon-Keller heads up co-innovation projects for SAP Africa and, located in France, Jean-Christoph Pazzaglia directs product management for the higher education team at SAP. Last fall, they met Dr. Taylor, while participating in Africa Code Week and began talking about how SAP HANA could help propel SKA – one of the South Africa’s premier science and technology undertakings.

Elke says, “Making sense of the huge amounts of data that the telescope will generate will be critical to the success of the research. If carried out correctly, this project has the potential to become the universe’s strongest validation case for SAP HANA.”

She continues, “To put this into context, the SKA project in South Africa will produce two terabytes per second of raw data. That is roughly enough data to fill 340,000 laptops with content a day.”

This data has to be transferred to the end user, reduced, calibrated and transformed into a 3D cube. Afterwards, the data must be analyzed and archived - a challenging and time consuming process.

International Task Force Shoots for The Stars

Meanwhile, back in Waldorf similar initiatives were underway. Klaus Schimmer works in the global security team at SAP but has a special interest in astronomy. In 2013, Klaus met with world re-known astrophysicist Professor Thomas Henning, director of the Max Planck Institute for Astronomy in Heidelberg, and an expert in star and planet formation. They began discussing how SAP HANA could be used to address the big data challenges presented by the massive telescopes like SKA and the Atacama Large Millimeter/submillimeter Array in Chile, where Dr. Henning worked.

The next year one of Dr. Henning’s former students, Tatiana Vasyunina, joined SAP and formed a team with Klaus to determine how SAP big data technologies could advance astrophysics. Recently, Franz Färber, the head of SAP HANA VORA, agreed to support the project.

Franz said, “These new telescopes create an almost unimaginable amount of data. This is a real challenge and we want to know how far we can go with our technology. Let’s see what SAP HANA Vora and SAP HANA can do in this situation.”

Virtual Pow Wow

All of these various activities culminated in a two-day workshop held in early March to explore how SAP could support the SKA’s big data challenges. This virtual meeting brought together SAP executives with executives and research experts from IDIA and the SKA project located in Cape Town, France, Heidelberg and Waldorf.

The Cape Town team held a design thinking workshop to refine the use cases for SAP big data technology. In April and May, technical teams from SAP, IDIA and SKA and SAP will further develop prototypes during in-person workshops.

This project is still in its exploratory phase. But the discussion about how SAP can help the universe run faster have begun.