Florida audit of COVID-19 data finds gaps in number of cases, deaths reported

TAMPA, Fla. (WFLA) — A review of Florida’s COVID-19 case and mortality statistics produced by the state’s Auditor General found the number of cases and deaths reported by the Department of Health were underreported due to data collection issues.

The report, published on June 1, comes after a Florida Dept. of Health Office of Inspector General investigation concluded claims by self-styled whistleblower Rebekah Jones that the state had modified data to downplay COVID-19 numbers in the state were false. NBC News obtained a copy of the OIG report in May.

The report from the FDOH OIG found the claims by Jones were “unsubstantiated” and “unfounded.”

The findings concerning Jones’ whistleblower allegations, that FDOH was directed to falsify positivity rates, that new case data on positivity was misrepresented, and that Jones was directed to restrict access to underlying data, were all reported false in the OIG report.

Now, the state’s Auditor General has found the case and death data for COVID-19 from FDOH was inaccurate, to some extents.

“To evaluate the State’s readiness to provide essential information needed to respond to the global pandemic, this operational audit focused on COVID-19 data collection and reporting processes at the Agency for Health Care Administration (Agency), Department of Health (Department), and Division of Emergency Management (Division) during the period March 1, 2020, through October 9, 2020,” the report said. “As subsequently described, the number of entities reporting data, apparent inaccurate or incomplete data reported to the State by those entities, and the lack of effective access controls in the systems used to gather data, impacted the State’s ability to accurately report COVID-19 data at the beginning of the pandemic.”

Detailing the Auditor General’s findings, the state report found that the “Merlin system,” a data software used by the Florida Health Department, contained some data that “did not appear complete or contained anomalies,” affecting its accuracy and “usefulness.” Data held in the Merlin system compared to data provided to the Bureau of Vital Statistics were different, according to the audit report.

“We compared Merlin death records to Bureau of Vital Statistics death records where COVID-19 was included as a cause or contributing factor of death and identified differences between the records,” according to the report.

Additionally, the department records “did not always evidence that COVID-19 positive individuals were contacted, or timely contacted,” against the requirements of the FDOH’s contact tracing guidelines.

“Neither the Division nor the Department reconciled the reported number of COVID-19 tests administered at state-led testing sites to laboratory results reported to the Department,” the audit reports. “Additionally, Division records did not always evidence that the Division reconciled the number of COVID-19 tests invoiced by laboratories to the number of tests reported in Division situation reports.”

The report also said agency records did not provide evidence of the state evaluating data accuracy, as far as COVID-19 data reported by hospitals, nursing homes, and assisted living facilities.

As the audit report only covered the period of March 1, 2020 to Oct. 9, 2020, the auditor wrote in the report that data recorded after that period “may be subject to future audit.”

Depending on the specifics of why a case or death was not included in Florida’s Vital Statistics records, thousands of death records may not have been included. The Florida auditor general’s analysis concluded that 2,495 death records that were reported in Merlin were not included in the Vital Statistics reports, while 3,082 death records in Vital Statistics were not included in the Merlin data.

Reasons for lack of inclusion in the Vital Statistics reports ranged from issues with “systematic match of the individual’s full name and date of birth” to “death records in the Vital Statistics data…could not be systematically matched to death records in the Merlin data based on the individuals’ full name and date of birth.”

Lack of match-up was among the reasons some deaths were not included in the data published by FDOH. The health department told the auditor general that some cases did not include lab results with COVID-19 either.

Additionally, some of the individuals who were recorded as having COVID-19 for a contributing cause of death did not have their records reported in the Merlin system used by FDOH. Others had no record of a testing positive for COVID-19 in Merlin, despite having it reported in the state’s Vital Statistics records.

“Absent complete and accurate information related to the extent and severity of the COVID-19 pandemic, government officials and the general public may not have had all the information necessary to assess the efficacy of COVID-19 control measures and take appropriate actions,” the auditor report read.

When the audit was in progress, the auditor general said “Department management indicated that, if the Department did not receive a positive lab result or if the name and demographics listed on the laboratory report differed from the death certificate, the Department may not have been able to match the record with enough confidence to definitively say that the individual who passed away was the same individual listed on the COVID-19 test result.”

Individuals that did not have records in Merlin, or who may not have received a laboratory test result confirming COVID-19, were not included as a COVID death in the Merlin system, according to the auditor general report.

Data for the number of cases of COVID-19, independent of the death records, also had “data quality issues.”

Some case records did not include residency information, gender, racial demographic information, age, or contained “invalid city names,” or had entries with incomplete information, including “numbers or entries of only one or two characters.”

When asked by the auditor, FDOH management told them “data quality issues were due to having to rely on data being sent from the laboratories, along with the large number of cases and limited resources to address data accuracy and completeness issues.”

The auditor general also evaluated FDOH efforts to make sure the data was accurate and complete in Merlin. While FDOH did attempt some accuracy checks, the auditor general found “7,718 instances where the dates of follow-up contact attempts were missing from Merlin,” as of the data reported through Oct. 9, 2020.

CHDs, the staff responsible for maintaining data on these attempts, were verbally notified to fix the problem. However, “the Department did not set a deadline for resolution” and “did not routinely perform data analyses to assess the completeness of all reported test results.”

Inquiries regarding contact tracing conducted by the state showed that despite Department of Health guidelines, there were multiple instances of contract tracing not occurring, or lacking documented follow-ups.

The auditor general found “for 168,880 of the 729,552 cases, Merlin did not evidence that the COVID-19 positive individuals were either contacted or contact was attempted by the Department.”

When asked about contact tracing follow-ups, and incomplete contact methods for doing so, the FDOH told the auditor general that “inaccurate contact information and the refusal of some individuals to speak with the department hindered” its “ability to contact all COVID-19 positive individuals.”

FDOH also told the auditor general that, as COVID cases spiked, data quality issues and discrepancies occurred.

In response to the audit report, State Surgeon General Dr. Joseph Ladapo wrote that the department agreed with the findings and a corrective action plan was put in place and is currently in progress, but noted that some discrepancies were to be expected.

“If COVID-19 testing was not performed, occurred more than 30 days before the death, was not reported to the Department, or could not be matched to a Vital Statistics record because of data quality deficiencies, the death would not be counted as a COVDI-19 associated surveillance death included in the Department’s COVID-19 surveillance reports,” Ladapo wrote in response to the auditor general. His response also noted that “most data quality issues the Department experienced during the COVID-19 pandemic originated from laboratories that submitted inaccurate or incomplete data” to FDOH.

He also wrote that the FDOH no longer “recommends” that county health departments perform contact tracing for every COVID-19 case, and that as of Jan. 11, the Bureau of Vital Statistics is still monitoring investigations in high-risk congregate settings every week.

“Conducting routine case interviews and contact tracing for all COVDI-19 cases is no longer an objective for the Department, nor is it part of the Department’s guidance,” Ladapo wrote.

Responses to the auditor from the Agency for Health Care Administration and the Department of Emergency Management, both made mention of challenges to monitoring and reporting during the pandemic. Kevin Guthrie, FDEM Director, did not provide a corrective plan, while Simone Marstiller, director of AHCA, included one in their response to the report.

Overall, response by state agencies to the report on data accuracy was to make changes to processing efforts, but noting the difficulty of data collection at the height of the pandemic. Improvements to IT access controls were mentioned as a way to ensure management of data remained possible for staff.