Negligence, mismanagement blamed for server crash, data loss at Government Printing Works

Poor control systems, gross negligence and top management that had no idea what they were doing are to blame for the collapse of a server and crucial data loss at Government Printing Works (GPW), a panel report into the February 2021 incident found.

The data lost after the crash cannot be recovered and has to be reconstructed from scratch, panel member Papati Malavi told MPs on Tuesday.

The panel was appointed by Home Affairs Minister Aaron Motsoaledi to probe the server crash that caused work at the entity to come to a standstill, affecting information dissemination, such as the government gazette. The website went offline and online again for more than a month.

ALSO READ: Home affairs suspends four officials linked to passport fraud syndicate

Motsoaledi assembled the team, led by former adviser to ex-president Thabo Mbeki, Mojanku Gumbi, after the minister received information from a whistleblower that the crash was not an accident, but a deliberate act.

“That worried me because GPW is a national keypoint. It worried me when legal professionals were unable to work, couldn’t finalise estates because of unavailability of documents… that amounted to holding the country’s economy at ransom,” said the minister.

Presenting the outcome of the probe to Parliament’s home affairs portfolio committee, Malavi said the former chief information officer Anele Apleni had been warned since 2019 about the danger of not migrating data from the damaged Hewlett Packard Enterprise (HPE) Virtual Array (EVA) server.

During the investigation, the panel was informed by ICT staff members that the crash was caused by a surge in electricity when power resumed after load shedding.

Upon contacting Eskom and the City of Tshwane to verify the information, the panel found out that there were no power cuts during that period. Non-compliant electrical installations at Pavillion 2, which housed the server, had caused the crash.

“The key direct finding is that the incident of 4 February 2021 was caused by poor maintenance of the ICT infrastructure due essentially to the fact that the CIO and his team did not know how to perform proper functions on the server – such as loading discs, scrubbing them before loading new data, ensuring that there is proper backups should there be a problem, because ICT equipment does fail.

“All of this was accompanied by a lack of support and maintenance contracts with service providers for the servicing of ICT-related equipment. Underpinning these issues, however, is a failure of management and supervision at various levels.”

There was no backup system in place for the server, Gumbi said.

“When the investigation commenced four months after the incident, the equipment and server had already been taken out of the country by the manufacturer.

“The information we had depended on expert evaluations given to us, and they said the reasons the hardware failed was because discs were not inserted properly or cleaned before being loaded. We could not make an independent assessment of whether someone interfered with the server or not.

“A CIO is part of senior management. In my view, when you are CIO, the minute you wake up you make sure you have backups running almost daily because machines crash, so you make sure backups are up to date. That did not happen in this case.”

Apleni resigned in January this year after being interviewed by members of the panel.

READ MORE: Cabinet condemns attacks on Motsoaledi amid Zimbabwe permit battle

‘Toxic relationship among top managers’

According to the investigators, a toxic atmosphere between senior managers and former acting CEO Thandi Moyo prevailed at GPW. Apleni told the panel that he had never approached Moyo whenever he experienced issues in his department.

“There were no formal meetings by executives. There were no meetings around 2017 and 2018. The secretary told us that there were no exco minutes prior to 2019… that indicates the total collapse of governance.

“There was no adherence to auditor-general rules to submit financials. It was difficult to get information from GPW, we insisted repeatedly for critical information and even wrote them a letter that we would give them an adverse finding if they didn’t produce information we requested,” Malavi said.

The permanent loss of data added to the outstanding financials required by the auditor-general to perform audits. Audit reports for the past two financial years are outstanding.

Under Moyo and former CFO Josephine Meyer, there were tender deviations until 2020.

“When asked why, because there was no original contract, no answers came forth. For the current CEO, problems started when she started asking for the contracts upon arrival at GPW. There were also clear duplications of duties as Meyer signed contracts as CFO and acting CEO [after Moyo left]. We asked why she did that… she said there was nowhere it was stated that it wasn’t allowed… she didn’t see anything wrong with that,” said Malavi.

Action against Moyo, Apleni recommended

Gumbi and her team recommended action against Apleni, Moyo and other senior managers for gross negligence. As a result of Apleni and Moyo having left GPW, the panel recommended that the Department of Public Service take note that the duo had outstanding issues should they pop up anywhere in government and its entities.

The probe found “reckless disregard” of GPW business continuity by Moyo for allegedly approving the business case for the termination of the contracts of IT service providers “who provided the outsourced ICT skills and thus putting GPW’s business continuity at risk.”

“Should she still be employed in government, the matter can be referred to the relevant accounting officer for consideration. The same should apply to all those who recommended such approval,” the panel said.

It also recommended that a ·contingency plan be put in place at GPW, including the procurement of the critical skills through the State Information Technology Agency (SITA) until the approved ICT posts are filled.

Motsoaledi told MPs he would personally oversee the implementation of the panel’s recommendations, adding that some of the wrongdoings had already been reversed.

“There is no instrument or method available in the open market to recover lost data. Current CIO Mr Van der Merwe was tasked with reconstructing this by the CEO. Van Der Merwe battled to try get the original contracts… the 2021 financials were reconstructed and submitted to the AG, at the moment, he is finalising reconstruction of the 2022 financials,” he said.

He said Apleni left immediately after he was confronted with evidence showing he lied about the server incident.

“I became suspicious immediately after being interviewed by the panel, I wanted to stop him from leaving GPW but was told I have no legal grounds to stop someone who wants to leave an entity. What is left is to find out what our recourse in law is against someone who causes damage and leaves.

“The panel did not find a smoking gun of sabotage, but found gross negligence. In my view, gross negligence could be a form of sabotage. I made that statement on the strength of someone working there telling us it wasn’t an accident. It seemingly was a known accident that was waiting to happen. The CIO was aware that the system was one foot into the grave, and it happened.”

According to her LinkedIn profile, Moyo was appointed deputy director-general of rural development at the Department of  Agriculture, Land Reform and Rural Development in March this year.

Probe reveals negligence, mismanagement at govt printing works

NOW READ: Motsoaledi advises Zimbabwean Exemption Permit holders to ignore ‘false hope created by HSF’