One of the key drivers behind the decision to migrate to the cloud has been the promise of lower costs. But as many organisations are fast discovering, not only are these cost savings failing to materialise, in some instances it can lead to costs soaring. 

A case in question is NASA’s recent decision to choose AWS (Amazon Web Services) to handle its Earthdata Cloud, the data repository for the Earth Science Data and Information System (ESDIS) that collates all the information collected from its various missions.

Previously, NASA was storing all its data on-prem across 12 DAACs (Distributed Active Archive Centres). But with no less than 15 missions planned over the next few years (with each expected to produce 100 terabytes of information every single day) NASA was faced with the prospect of their data growing from 30 petabytes to over 250 petabytes by 2025. 

So, after lengthy and supposedly exhaustive consultations, last year NASA chose AWS (Amazon Web Services) to handle its repository for the data collected from all future missions.

“Houston, we have a problem.“ 

Unfortunately, someone at the Agency forgot to take into account the associated costs (a.k.a. the egress charges) of retrieving the data they feed into AWS. 

These egress charges are the costs incurred when transferring data from the Cloud to another area, which in the case of NASA could simply be a local workstation for one of their engineers or scientists. The vast majority of AWS subscriptions will charge these fees over and above the agreed monthly Cloud subscription, so the more data you retrieve, the bigger the bill. 

At the moment, when users download data from a DAAC, there are no additional costs above the need to maintain the existing infrastructure. But when users download from the Earthdata Cloud, NASA is charged every single time, while still having to maintain the 12 DAACs. 

To add to an already messy situation, ESDIS hasn’t yet determined which data sets will migrate to the Earthdata Cloud, nor has it developed cost models based on operational experience and metrics for usage and egress. Consequently, current cost projections may be far lower than what will be necessary to cover future expenses and Cloud adoption may well become increasingly expensive and difficult to manage.

Should NASA limit the amount of data in order to control costs, the result could be that valuable scientific data becomes less available to users, thereby negating one of the key reasons for migrating to the Cloud in the first place. 

Either way, NASA is facing a bill of astronomical proportions. 

Enter the Inspector General. 

These revelations have caused huge consternation throughout the Agency, and as a result the office of the Inspector General of NASA undertook a full audit of the project. 

It concluded that the people in charge of the Earth Observing System Data and Information System (EOSDIS), which makes available the information from ESDIS, had simply failed to consider the additional costs of the eye-watering egress charges. 

The report also highlighted the fact that the rather embarrassingly named Evolution, Enhancement, and Efficiency (E&E) panel that was chosen to review the DAACs, didn’t even attempt to identify potential cost savings. 

To add to an altogether shambolic situation, the panel also failed to adhere to the National Institute of Standards and Technology’s (NIST’s) data integrity standards, displaying a complete lack of independence as half of the panel members also worked on ESDIS. 

The report concluded that once 2 key projects are up and running and providing sufficient data, a comprehensive and truly independent analysis should be conducted in order to determine the long-term financial implications of supporting Cloud migration, while also maintaining the existing DAAC footprint. 

Heads stuck in the Cloud? 

Putting the blame to one side, many questions remain. 

Some are asking if NASA should now make a comparative evaluation of the cost of upgrading its DAACs to meet the 250+ petabytes storage requirements versus the migration to AWS with the egress charges fully factored in. 

It may well be too late as NASA are already committed to AWS, but with the very long-term future in mind it’s worth noting that there are many who believe that while the Cloud is great for bursty/on-demand workloads, when it comes to constant load then on-prem is better, safer and usually works out cheaper. 

Down to earth advice and support are available at Quadris. 

Migrating to the Cloud may appear straightforward, but as you can see even organisations such as NASA can make errors of judgement when confronted with new technologies and operating practices. 

For an in-depth discussion about the pros and cons of Cloud migration v On-prem, and the different options available to you, contact Peter Grayson on 0161 537 4980 or email peter.grayson@quadris.co.uk