Prior to testing of software and hardware as planned during February new database servers were purchased.
Performance and capacity tests were repeated in the same way as in previous tests in November with the software, hardware and set up of servers that was used at 25manna 2013. The same response time issues that arose during the competition remained.
Tests were carried out with the new servers, with the same software and set up that was used at 25manna 2013. The tests were then able to be completed without performance issues.
Conclusion:
Why was there a problem with the results?
The problems during 25manna 2013 were caused by a new version of the database engine MySQL installed too close to competition day, without new performance tests being conducted.
The new version of MySQL uses significantly more RAM than the old one, therefore the servers were overloaded. Since all servers had the same software versions and memory configuration the same problem affected the backup servers.
New performance testing was not conducted prior to 25manna due to a combination of that no one could imagine that a MySQL update could have such devastating consequences and that the change was introduced so close to race day.
Was it the Sportident and/or OLA systems that did not work?
No, the problem was not caused by the Sportident or the OLA systems.
What measures are taken?
The servers used during 25manna 2013 will be scrapped.
No new software will be introduced without capacity tests being carried out.
How can we learn from what happened for 25manna and other competitions in the future?
As part of the process of performance and capacity tests, the tests have been supplemented in order to provide as complete testing of equipment, with associated software, as possible.
Work is continuing to develop methods etc. to try to prevent similar events occurring in the future. In parallel, work is underway to improve the organization and processes to use if/when something similar were to occur again, in spite of the improved testing and methods.
