This story was updated at 1:02 p.m. on Thursday, July 2, with a statement from VMware.
In a new seven-page explanation, the Hamilton County Sheriff’s Office is blaming two companies it uses to maintain its dashboard camera footage for a server failure that affected 15 months worth of video.
The statement follows the federal court filing of an affidavit authored by District Attorney General Neal Pinkston in which he accused the sheriff’s office of a lack of cooperation with the criminal investigation into former deputy Daniel Wilkey, who faces 44 criminal charges and 10 lawsuits.
In its explanation, the sheriff’s office denied Pinkston’s claim that the Tennessee Bureau of Investigation had to step in before Wilkey’s dashboard camera videos were turned over to the DA’s office.
With the server failure, the sheriff’s office says it was caused by “a combination of several design factors of the two separate vendors” — VMWare and Veeam.
In a statement on Thursday, VMWare denied the data loss was caused by its software.
“VMware support team has concluded the data loss was not the result of a product defect within VMware vSphere,” the statement read. “Hundreds of thousands of enterprises trust VMware as their digital foundation to offer services for their customers and citizens, and we are not currently seeing this issue with other customers.”
Veeam Software declined to comment.
Sheriff’s office information system staff realized there was a problem in January, when they noticed the nearly 13-year-old server was moving slowly, the Times Free Press reported previously.
At the same time, Pinkston’s office had been weighing more criminal charges against Wilkey.
“When I informed the sheriff’s office I would be seeking other videos, I was told there had been a catastrophic failure of their computer server and they had no ability to recover any dash cam videos from a 15-month period including the first seven months of 2019,” Pinkston wrote in his affidavit.
Sheriff Jim Hammond has denied any purposeful act to cause the server to fail.
According to its explanation, as the sheriff’s office tried to identify the problem, then discovered the software hadn’t been properly backing up copies of the drives.
Copies, or “snapshots,” of a drive taken at certain intervals capture what that drive looks like at that specific moment. As new snapshots are taken, a program is used to compare the old ones with the new ones, consolidate them and discard any unnecessary snapshots. Those consolidated snapshots can later be used to recover lost data, at least up to the time the last snapshot was taken.
According to the sheriff’s office, snapshots hadn’t been successfully consolidated since Dec. 31, 2019, leaving too many snapshots on the server and causing it to run slow.
“The initial assessment was that the [server] was running with 92 snapshots piled up on the disk,” according to the sheriff’s office statement. “Per VMWare’s specifications, any machine holding more than 30 snapshots would be running a high risk of data corruption.”
The sheriff’s office reached out to the operating system vendor in an attempt to remedy the issue and followed its instructions, but that resulted in a corruption of all data.
Specifically, the sheriff’s office blames VMWare’s design decision to not notify users of the number of existing snapshots without the user creating a customized alarm. And Veeam’s design decision to treat unsuccessful disk consolidation as an “informational event instead of a warning or error.”
In an attempt to recover the lost footage, the sheriff’s office moved the data from the server to a disk drive to send to a data recovery company.
According to Wednesday’s statement, sheriff’s office personnel then “re-format[ted] the existing video drive space on the [server] in order to set up a new clean and working version of the [server] and storage space for the continued operation of our officers’ in-car video footage that was still being generated on a daily basis and waiting for upload.”
HCSO’s statement regarding dash-cam server crash
Reformatting is the process of preparing a disk to be used for new data by erasing all information on a drive.
No second copy of the corrupted data was created because it would take 10 days to make a new copy and it would “run the risk of failing on fire up,” Ron Bernard, director of information systems for the sheriff’s office, told the Times Free Press in February.
That singular copy was then sent to the data recovery company in California in early February, Bernard has said. By Feb. 17, the sheriff’s office “received the news that 99% of the data” was “essentially useless.”
So the disk containing the singular copy of the corrupted data was shipped back to the Hamilton County Sheriff’s Office, “where it was promptly placed into secure storage inside our Property and Evidence facility for safekeeping, where it has remained since that time.”
The server failure came amid several federal lawsuits, most of which had still not gone through the discovery phase, which is when attorneys review evidence they’ve requested.
The same day the Times Free Press reported the data loss, attorneys representing the alleged victims of former deputy Wilkey filed motions in federal court asking a judge to order the county to preserve all electronic evidence and to allow the plaintiffs to conduct a forensic examination of all technology involved with preserving the videos.
The judge would later order county attorneys to file weekly reports to determine which videos still existed. A decision on whether a forensic examination will be allowed is still in the air.
In one of those reports, county attorneys noted “some potential discrepancies to be addressed” and disclosed that more videos had been found, including 12 additional videos that were turned over the the DA’s office.
Now, the plaintiffs are seeking sanctions against the county because it reportedly disseminated those videos to lawyers for the accused deputies and to the district attorney’s office but not to the lawyers for the plaintiffs until after those lawyers asked for copies.
Attached to the motion for sanctions was Pinkston’s affidavit detailing the difficulties he’s had in obtaining evidence from the sheriff’s office in the criminal investigation of Wilkey, who stands accused of six counts of sexual battery, two counts of rape and nine counts of official oppression.
For example, when requesting Wilkey’s in-car videos, Pinkston said Chief Deputy Austin Garrett told him “it would be another 120 to 180 days before we would receive the Wilkey videos.”
That was an “inordinate amount of time to copy all the videos,” Pinkston has previously said. So he contacted the Tennessee Bureau of Investigation, and two days later, an agent arrived in Chattanooga to retrieve the videos.
By 1 p.m. that same day, the agent was able to copy 578 videos recorded by Wilkey’s dashboard camera.
“Yes, the TBI did come in, but guess who showed the TBI how to do it and help them get it done? The sheriff’s office. Did you see that mentioned? No. All the TBI did was come in here and lend their expertise in helping us get him what he wanted in his time frame,” Hammond told the Times Free Press on Monday.
On Wednesday, however, the sheriff’s office said the TBI never “participate[d] in the pulling of videos.”
“In fact, with the assistance of the vendor, the HCSO Network staff was able to utilize a mass export solution, so that an initial copy of all videos requested was delivered to the DA’s office by the time that the TBI agent had arrived onsite at the HCSO,” the statement read. “The TBI’s only contribution to the project was supplying six (6) extra laptops to the District Attorney’s Office, as well as to assist in getting the contents of the original removable drive copied to said laptops, in order to facilitate review by more than one individual at the DA’s Office.”
Pinkston’s office declined to comment, stating that the affidavit speaks for itself.