Page Comparison

This project is about delivering fit-for-purpose storage and computing infrastructure. This will be delivered by deploying new hardware, software, and expertise to create: 

1. A central storage space for all AgResearch’s research data. 

   • Replacing our aging legacy High Performance Computing (HPC) with a more performant compute environment 

2. Provide application support to researchers through our partnership with the talented people at NeSI (New Zealand eScience Infrastructure) 

Relevant Gateway articles: 

https://agresearchnz.sharepoint.com/sites/Gateway/SitePages/eResearch-platform-update.aspx  

https://agresearchnz.sharepoint.com/sites/Gateway/SitePages/Green-light-for-eResearch---our-first-Enabling-Platform.aspx  

eResearch enabling platform has four objectives to improve AgResearch’s eResearch capability. One of those objectives is to provide fit-for-purpose storage and computing infrastructure.  

A central storage space will help with: 

• Data discoverability - the central storage space will work with our new Outputs Management System (OMS) to ensure all research data are catalogued, hence assisting with data discoverability. 

• Data organisation - this will make it easier to locate and access data. Additionally, it will ensure that data is stored in a consistent and secure manner. 

• Data protection - there will be robust systems in place for backup and recovery thus protecting valuable research data from loss, corruption, and malware. 

• Collaboration - the system will allow us to give access to our collaborators, thus enabling researchers to collaborate on projects more easily.

The HPC will enable our researchers to: 

• Processing large data sets, thus enabling them to analyse and visualize their data in a timely manner.  

• Developing, training and scaling up models in Machine Learning and Deep Learning. 

• Modelling and simulating complex systems 

• Collaboration: The new system will enable our collaborators to also access the data and compute system. This is especially important because most of our research now is moving towards transdisciplinary fields and we will need expertise from a range of areas to to tackle complex problems.  

If you work with research data at AgResearch things will be changing for you.  

Data will be stored in a different place 

As mentioned earlier, one of the main components of this project is centralising all our organisation’s research data. As your data is moved to the new infrastructure file paths (i.e. the links you use to access data) will break. To minimise the impact to you we will be doing the migration for you, communicating throughout the change and will be ready to help if you need it.

Data will be stored in ‘Projects’ or 'Datasets

Absolutely, we will work with you and plan the data and workflow migration. For time critical projects, our priority will be to minimize any disruption. This will mean you can continue on working on existing HPC infrastructure to finish your project and we will migrate your data and workflows some time well after 30 June. 

Our compute and primary data storage is located within NeSI’s Flexible HPC platform at Waipapa Taumata Rau’s (University of Auckland’s) Tāmaki Data Centre in Tāmaki Makaurau, Auckland. Geographically distinct back-up copies of the data are being made on AgResearch Infrastructure at NIWA’s High Performance Computing Facility (HPCF) Data Centre at Greta Point, Wellington.  

We are not intending to charge for use, it will be treated as overhead. 

We do intend to account for all use, both compute and storage to build a picture of how the infrastructure is utilized and by whom. 

The one caveat is around fair use, if there is going to be a significantly large request for resource we may ask for a capex contribution to extend capacity. Ideally these sorts of requests go through the eResearch Platform Advisory service so they can be picked up before funding has been allocated. 

We will support standard growth for expansions so standard growth can be addressed, once we get data to forecast how we are tracking. We are hopeful 3PB will give us a good starting point for the storage infrastructure. 

While we will not be charging our internal users/projects for using the infrastructure, if you would like to pass some/all costs for use of the infrastructure through to your external customers we will have a mechanism for understanding usage and will be developing pricing as the need arises.