Is the AFM doing a good job?

‘Freaky Thursday’ has definitely changed the Dutch audit sector. For those people who lived on another planet during the last months: on Thursday September 25, the Autoriteit Financiele Markten (AFM), which is the Dutch regulator that has to monitor the audit and financial sector, published a report in which it is mentioned that 3, 4, 4, and 7 out of the 10 investigated audit engagements of respectively EY, PwC, Deloitte, and KPMG were of insufficient quality. Based on these results, the AFM concludes that the audit quality of the Big4 auditors in general is insufficient and that no improvements have been made since the previous investigation in 2010.

Three Major Weaknesses

Although the fraud cases of the last couple of years seem to suggest the existence of structural problems in the Dutch Big4 audit firms, the only valid conclusion I can draw after reading the AFM-report is that the AFM missed a historical opportunity to reveal these structural problems in a credible way. Indeed, the investigation of the AFM is not worth the name ‘research’ and violates the basic principles of how reliable research should be conducted. The weaknesses in the AFM-report are at least threefold. First, the AFM has only investigated 10 audit engagements for every Big4-auditor. This sample size is too low for making general statements about the overall audit quality of the Dutch Big4 auditors. My recommendation is to use at least 50 audit engagements for every Big4 auditor. Second, the sample of the AFM is not randomly chosen which implies that the observed audit quality in the AFM-sample is not a reliable source of information to make reliable conclusions about the general audit quality of the Dutch Big4 auditors. Auditors are often accused for taking samples that confirm their initial hypotheses. Based on the way in which the AFM-investigation has been conducted, one can argue that this critique also applies to the AFM. Third, as the AFM only gives very limited information about the way in which this investigation has been conducted, it is impossible to know whether the results can be used to make conclusions about the differences in audit quality of the Dutch Big4 auditors. Such a comparison between the Dutch Big4 auditors is an important purpose of the report as the AFM states the following in the report: “….users of this report, such as investors and creditors, can see how audit quality of the different auditors differs from each other…”. Comparing the Dutch Big4 auditors based on the results of the AFM-investigation is impossible because it could well be the case that KPMG has more mistakes in its 10 audit engagements because the 10 audit engagement of KPMG are more complex than the chosen audit engagements of the other Big4 auditors. Also the conclusion that the audit quality of the Dutch Big4 auditors has not improved since 2010 can be rejected based on the same argumentation. A possible counter-argument of the AFM could be that the complexity of the audit engagement does not matter as audit engagements should always comply with the legal rules. This counter-argument is not really persuasive as the AFM has on purpose investigated more complex audit engagements. The AFM is thus aware that insufficient audit quality is more likely to be observed in more complex audit engagements.

Again, my argument is not that the audit sector does not suffer from structural problems, but that the investigation of the AFM cannot be used to conclude that the general audit quality of the Dutch Big4 auditors is low. The conclusion of the AFM about the insufficient audit quality further loses persuasiveness as the AFM also concludes that, based on another investigation, the overall quality of financial reporting of Dutch firms has increased. Just as my colleague at Tilburg University Jan Bouwens, I am really wondering how an insufficient audit quality can be combined with an increase in the quality of financial reporting. As the auditor has to control the financial reports, the quality of financial reporting can only increase if the auditor is doing a good job.

Good news, Bad News

The way in which the AFM tries to reveal the structural problems in the Dutch audit sector, can be compared with a lawyer who tries to prove a crime based on wrongfully or illegally collected evidence. Although such evidence can indicate that the crime has taken place, the value of wrongfully collected evidence is zero and cannot be used to determine the penalty. Applying this legal logic to the AFM-investigation implies that the AFM-investigation cannot be used to make conclusions about the general audit quality of the Dutch Big4 auditors. As the weaknesses in the AFM-investigation do not require higher-order research skills, it is very remarkable that a lot of institutions and individuals nearly literally copy-paste the conclusions of the AFM. One expects a more critical mindset from institutions and individuals who use the AFM-report to accuse the auditor of lacking a critical mindset. The Big4 auditors cannot be blamed for missing a critical mindset about the AFM-report because they are currently the object of a heated discussion. The good news of this heated discussion that is currently going on is that we can expect some changes in the Dutch audit sector that will address some of the structural problems. The bad news is that the fuss of the last months has caused (or reinforced) serious doubts about the way in which the AFM fulfills its monitoring role. It sounds like a good idea for the AFM to critically think about the way in which they want to monitor the Dutch audit sector in the future! Just as has been argued for the audit firms, I think it would be good if the AFM could come up with some proposals to improve its own functioning!