“Insufficient technology resources” and a lack of “robust tools” brought New Zealand’s stock exchange to a halt multiple times last year, a new report by NZ’s financial watchdog has found.

The Financial Markets Authority (FMA) audit was launched in the wake of multiple systems outages, including one major market outage, in early 2020; the scope of the audit was subsequently expanded to examine major distributed denial of service (DDoS) attacks occurring in the latter half of 2020.

According to the FMA, the performance of NZX’s systems failed to meet regulatory requirements or “expectations for fair, orderly and transparent markets”.

NZX’s internal IT capability was slammed by the watchdog as being “consistent with a small to medium-sized New Zealand corporate, but not of a standard expected for systemically important infrastructure”.

“The technology was largely managed in-house, but the level of staffing was not sufficient to adequately manage mission-critical technologies, as well as provide appropriate risk management and security protection.”

At the time of the review, the report noted, “the team was mostly limited to day-to-day operations, defect management and making small incremental improvements.”

The FMA said that fundamental tools and practices were either “lacking, insufficiently robust or not fully utilised”, impacting NZX’s ability to ensure “a high quality of system health and resilience in the respects identified”.

Among the main causes of the outage identified by the auditor included a lack of: continuous version management and upgrades of supported software; performance monitoring and alerting; full testing lifecycle capability; capacity planning/management and fail-over capability; robust and recoverable processing standards; appropriate internal staffing, knowledge sharing and contractual arrangements to access external expertise.

NZX’s culture also came under fire, with the FMA lambasting the Exchange for failing to take “responsibility for known systemic and industry-wide issues” and not acting “quickly to remediate concerns raised”.

As a result of the outages, the FMA concluded that NZX ultimately breached its obligations under New Zealand’s Financial Markets Conduct Act 2013 (FMC Act), which requires it to uphold “the availability, security, capacity and maintenance of its trading platforms, settlement systems, internal market monitoring systems and other related systems”.

Overall, the FMA review concluded that NZX ultimately lacked adequate capability across its “people, processes and platform” to comply with these obligations.

Between March and April last year, NZX experienced nine significant technology-related incidents, including multiple volume-related system issues and at least one market outage.

“[These incidents] caused disruption and halted market activity, put investor confidence at risk, and negatively impacted Participants and the broader market ecosystem,” the FMA noted in its report.

This included disruptions to clearing and settlement, resulting in delays in processing, failures in messaging, and a requirement for data to be extracted and provided manually for reconciliation purposes (which erroneously had both missing and duplicate transactions).

A subsequent major DDoS attack in August 2020 resulted in issuers and investors being unable to use the main NZX.com site and the MAP (or Market Announcement Platform) to release and receive market announcements. This led to NZX halting the market for approximately four days “with only intermittent periods of availability”, the report noted.

The FMA challenged NZX’s view that such an attack was not foreseeable and could not be sufficiently prepared for.

“We consider a DDoS attack was foreseeable and that an attack of sufficient magnitude to take down the servers was at least possible and should have been planned for.”

It noted problematic and “rudimentary” crisis management planning, an organisational structure “missing” key IT security roles, and “suboptimal robustness of applications, poor network design and unprotected infrastructure”, among other faults in NZX’s security posture.

With regard to NZX’s trading volume-related issues, the FMA concluded that fundamental tools and practices were either lacking, insufficiently robust, or not fully utilised.

While the report conceded that trading volumes at the time “reached six times the daily average”, likely owing to pandemic-related trading spikes, the auditor noted that the Exchange “was running so close to the margins in terms of capacity that even a smaller unexpected volume spike may have resulted in similar issues”.

It stressed that the NZX should have been aware of capacity limitations of its core back-end processing system (known as BaNCS), “particularly as daily trading volumes had increased in the last three years”.

“Significant volume increases and DDoS attacks were… well-known risks that NZX had not adequately prepared for.”

“Many other exchanges worldwide have experienced significant volume increases and DDoS attacks but we have not seen any that were disrupted as often or for such a long period.”

Warnings from the system vendor in late 2019 that the system was nearing capacity went unheeded, the FMA states. However, NZX contends this information was “never communicated to them” by the vendor.

Cultural problems

FMA chief executive, Rob Everett, said feedback from market participants also underscored significant cultural issues at NZX, particularly in its failure to accept responsibility for known systemic issues and its laggard remediation processes.

“The feedback from market participants mirrors our own observations and is a major concern that needs to be addressed by the NZX Board and Executive.”

“The failure to properly consider the broader ecosystem in which the exchange operates, and to fully engage with industry feedback and concerns, were contributing factors to the volume-related issues.”

Regarding the DDoS attacks, the FMA review found NZX’s crisis management planning and procedures were “basic” – which was at the very least acknowledged by the NZX.

“A DDoS attack was foreseeable, the FMA review found, and an attack of sufficient magnitude to take down servers – and with them, NZX’s market announcement platform – was at least possible and should have been planned for.”

“NZX self-rated its IT security profile at a basic maturity level, indicating that a number of best practices had not been adopted.”

The FMA’s full review report can be found here.