Data Pipelines at the Advanced Photon Source

Abstract: The Advanced Photon Source (APS) is an X-ray light source at Argonne National Laboratory where researchers perform imaging, diffraction and spectroscopy experiments at 68 specialized beamlines. APS recently completed a major upgrade that made it the brightest synchrotron in the world and upgraded many of the beamlines to take advantage of the dramatically brighter, more coherent beam. These new capabilities are increasing the facility’s raw data production from <10 PB/year before the upgrade to >100 PB/year once all instruments are back online. To meet this data challenge, the APS Beamline Data Pipeline project team is partnering with beamline staff to develop and deploy science data pipelines that use a common set of open-source software tools (many written in Python) to link instruments with the Polaris supercomputer at the Argonne Leadership Computing Facility (ALCF) for data processing. This talk will describe the APS data systems, our file-based and streaming pipeline templates, and some of the open-source software we use to extract scientific insights from beamline data.

Audience Takeaway: Open source Python data acquisition and analysis tools are ubiquitous at the APS. Many of the packages that are mentioned can be found at https://github.com/AdvancedPhotonSource.

This is a hybrid event. To attend online, join us on Zoom here at 6pm:
https://numfocus-org.zoom.us/j/89399976851?pwd=UEgMUZXdYmKdK1x1dIPL6hwUYnp7NW.1

Sponsor: Grainger will provide the meeting site, as well as snacks for the onsite participants.

• Address: 222 W Merchandise Mart Plaza, 18th Floor, Chicago, IL 60654
• Grainger's Overview: W.W. Grainger, Inc., is a leading broadline distributor with over $17 billion in revenue. At Grainger, We Keep the World Working® by serving more than 4.5 million customers worldwide with maintenance, repair, and operating (MRO) products and services delivered through innovative technology and deep customer relationships. Our machine learning team builds broadly across search, recommendations, product matching, computer vision, and generative AI.
• Logistics: To access the Grainger office, we require first and last names of those who RSVP'd by Feb 11. Please take the main elevators at the center of the Mart. Attendees will then present their IDs when they arrive at the front desk on the 18th floor.