Ten years of international collaboration on open source code
The Danish Meteorological Institute (DMI) and SMHI began working together on open source code for processing satellite data ten years ago. Today, this collaboration – Pytroll – involves more than one hundred experts and developers from around fifteen different countries, and the software has grown to become one of the most widely used platforms for real-time satellite data processing in Europe. Pytroll has enabled SMHI to process an ever larger volume of data for increasingly demanding external users.
Zoom imageA satellite view of Hurricane Ophelia in 2017.
Zoom imageThe short code that built the satellite image of Hurricane Ophelia.
Ten years ago, SMHI’s satellite experts Adam Dybbroe and Martin Raspaud travelled by train to Copenhagen for discussions with three Danish colleagues at DMI on closer cooperation when developing software for operational satellite data processing. They used the same programming language (Python), and had the same type of data and the same needs. Both organisations found themselves in a situation where modernisation was essential but there was a lack of resources. Another goal was to avoid solutions that would result in maintenance and development being difficult and costly.
Watch ten years of coding development in one minute
External link.
They began by meeting up every three months in Copenhagen or Norrköping, spending three or four days coding together. Interest in their work grew during 2011, with the Finnish and Icelandic weather services also coming on board.
“We held the first open workshop at SMHI in autumn 2012,” explains Adam Dybbroe. “In addition to SMHI and DMI, the workshop also included participants from the meteorological institutes in the Netherlands, Romania and Finland, and from EUMETSAT, the European Organisation for the Exploitation of Meteorological Satellites.”
Continued growth
Since then, Pytroll has continued to grow. Now, a number of Europe’s major meteorological institutions (including DWD in Germany and Météo-France) and EUMETSAT use and contribute towards the platform.
“The growing support and commitment we have received from EUMETSAT is particularly pleasing,” says Martin Raspaud. “Talented developers from across the Atlantic are now also involved, and Pytroll is now actively used in the USA and Canada.”
A high degree of activity
Pytroll’s global reach means that there is almost constant activity around the clock on pytroll.slack.com, where developers and users can hold discussions and help each other. The channel now has more than 300 members, with an average of around 60 active weekly users and approximately 200 messages each day.
Zoom imagePytroll’s tenth anniversary celebrations brought together developers and users for a hackathon, Pytroll Contributor Week (PCW).
“For several years now, developers and researchers have been meeting twice a year and spending a week coding together,” concludes Adam Dybbroe. “At last week’s hackathon, which now goes by the name of Pytroll Contributor Week (PCW), we celebrated ten years with Pytroll. This involved continuing to improve and development the code so that we can deal with EUMETSAT’s new satellite programme, in which the first satellite is planned to be launched in around two years’ time.”
Zoom imageEUMETSAT held its user days in November, at which a Pytroll stand was presented. SMHI’s Adam Dybbroe attended the event.
Pytroll: easy to use, modular, free and open code
Pytroll is the name of an international open source code collaboration for processing Earth observation satellite data. It is also used as an umbrella term for all the software modules developed or maintained as part of this collaboration. In developing Pytroll’s software, there has been a particular emphasis on ensuring that it is easy to use, modular, free and open. All code is written in Python, and it should be possible to use the software both for research and development and in a 24/7 operational production environment.
Pytroll began in autumn 2009 as a collaboration between DMI and SMHI on processing and using satellite data. From the start, there has been a focus on developing open source code to create a modern, flexible and robust production platform that can cope with growing demands, both today and in the future.
Sharing resources between several countries means that greater benefit can be achieved at the same cost. By making the source code free and open, the production systems will be better and more robust, and others will be more tempted to join the collaboration.
All Pytroll code uses copyleft licences (GPLv3 and LGPLv3), ensuring that all the code developed as part of the Pytroll project remains open. The code is available on GitHub, PyPI and conda-forge. Users and developers can interact on pytroll.slack.com and via Pytroll’s Google Groups mailing list.
There are now Pytroll users all around the world, and the code is used operationally in some form by around a third of EUMETSAT’s thirty member nations.
A total of 133 different developers have contributed to the code base of approximately 225.000 lines of code during the ten years of Pytroll’s existence, and on average there are more than 20 active monthly users.
According to the COCOMO model for developing large-scale software projects, Pytroll corresponds to 59 person years of development, and currently has a value of USD 3.2 million.
For more information, visit Pytroll.org External link..