- IN21B-0338: Sidecars in the Cloud: Metadata Stewardship for Large-Scale Scientific Archives
-
Board 0338‚ Hall EFG (Poster Hall)NOLA CC
Author(s):Generic 'disconnected' Message
James Gallagher, OPeNDAP, Inc. (First Author)
Miguel Jimenez-Urias, OPeNDAP, Inc. (Presenting Author)
NASA and other agencies are moving large amounts of Earth science data from traditional servers to the cloud, such as Amazon Web Services. This shift helps make data easier to share and use but also creates new challenges for managing and preserving it.Many of these data files were originally stored in ways that work well on local computers but not in the cloud. To fix this, NASA and its partners developed a new way to access the data efficiently in cloud computing systems. This new method requires an extra file—called a 'sidecar file'—for every data file. These sidecar files act like maps, showing exactly where to find the pieces of data in the cloud.
But managing these sidecar files is a big job. If a data file moves or changes, the sidecar file must also be updated, or the data won’t be accessible. There are also issues with inconsistencies in NASA’s shared metadata system, which stores information about all the files. These inconsistencies can make it hard to find or use the data.
While some of these problems are common in computer science, they’re new for many people managing scientific data.
Scientific DisciplineNeighborhoodType
Enter Note
Go to previous page in this tab
Session
