The explainability problem: when “the model said no” isn’t enough

A
Credit risk scoring · Bank statement analytics
AI system interface showing automated decision process

South African credit providers have been automating credit decisions for years. Scoring engines, rules-based approval systems, and increasingly ML-driven models now process applications at volumes and speeds that manual underwriting cannot match. The operational efficiency case is settled. The compliance case is less settled than many credit product managers currently assume.

The March 2026 court judgment on NCA Section 81 was unambiguous: the NCA’s affordability assessment requirement is substantive, not procedural. A credit provider who runs an application through a scoring model, compares the output to a threshold, and returns a decline has conducted a process. The court found that going through the motions does not satisfy the legal standard. The standard requires genuine assessment, and a genuine assessment requires that someone, at some point in the workflow, can explain the reasoning behind the outcome in substantive terms.

“The model said no” is not a reason. It is a description of a process that may or may not have involved a substantive assessment. The difference between those two things is the difference between NCA compliance and NCA exposure.

What the NCA actually requires when you decline

NCA Section 81 imposes an assessment obligation on credit providers before entering into a credit agreement. The assessment must consider the consumer’s financial means, prospects, and obligations. Where the outcome is a decline, the NCA requires that the credit provider be able to explain the basis for that decision in terms of the consumer’s actual financial position.

This is a higher bar than many automated systems are built to clear. A score below a threshold is the system’s output. It is not an explanation of the consumer’s financial position. The explanation requires: what income was verified, what obligations were identified, what expense norms were applied, what the resulting discretionary income was, and why that figure did not support the proposed obligation.

Courts distinguish between two types of compliance. Procedural compliance: the credit provider ran an application through a system that produces a score. Substantive compliance: the credit provider can demonstrate that the system’s output was based on a genuine engagement with the consumer’s financial reality, and that a human in the credit provider’s organisation understands and can articulate the factors that drove the outcome.

The March 2026 judgment drew that line explicitly. A system that produces only a score without supporting reasoning does not satisfy the substantive requirement. The judgment’s language, that “going through the motions satisfies nothing,” applies directly to automated decisioning systems that produce outcomes without documented, accessible reasoning.

The March 2026 court judgment: what it established

The March 2026 judgment arose from a challenge to a credit provider’s affordability process. The credit provider had an automated scoring system; the application had been processed through it; a decision had been produced. The challenge was not that no assessment was conducted; it was that the assessment was not substantive.

The court’s analysis turned on the nature of the Section 81 obligation. The NCA does not merely require that a credit provider conduct a process that produces an output. It requires that the credit provider assess (as a verb meaning genuine consideration) the consumer’s financial position. An automated system that converts inputs into a score according to a model it cannot explain to a human decision-maker is not conducting an assessment in the sense Section 81 intends.

The judgment has immediate implications for credit providers operating black-box models: models where the factor contributions to a decision output cannot be identified, quantified, or explained in plain language. If the credit provider’s underwriter cannot look at a declined application and explain, in financial terms, why the consumer was declined, with reference to specific verified income, specific obligations, and a specific affordability calculation, the substantive assessment standard has not been met.

This does not prohibit automation. It requires that automation be designed to produce explainable outputs and that human oversight be present in the decision workflow. The system scores; the human approves or declines with understanding of the reasoning. The audit trail captures both.

The POPIA angle: transparency in automated decisions

South Africa does not yet have a statutory right to algorithmic explanation equivalent to the EU’s GDPR Article 22. But the Protection of Personal Information Act creates a directional obligation that is moving toward greater transparency requirements for automated decision systems.

POPIA’s conditions for lawful processing of personal information include: processing must be adequate, relevant, and not excessive for its purpose; it must be transparent; and the data subject must be able to hold the responsible party accountable. In the context of automated credit decisioning:

The combination of the NCA’s substantive assessment requirement and POPIA’s transparency and accountability conditions creates a combined compliance posture for automated credit systems: the system must be explainable, and the explanation must be accessible to both the credit provider’s own review processes and, in appropriate circumstances, to the consumer.

Why black-box models are a regulatory liability

A black-box credit model, for this purpose, is one where the factors contributing to a specific decision outcome cannot be identified and quantified for that application. The model produces a score; the factors driving that score are either not available or not surfaced to the decision-maker. The credit officer sees a number, compares it to a threshold, and acts on the result without necessarily understanding why that number was produced for that consumer.

The regulatory liability this creates is not theoretical. The March 2026 judgment identifies it directly. But consider also the practical scenario: a consumer challenges a credit decline, through the NCR, the NCT, or litigation. The credit provider is asked to produce its affordability assessment for the application in question. If the response is “our model scored this consumer 412 against a threshold of 500,” the credit provider has described a process. It has not described an assessment. The question that will follow, and that the NCT will press on, is: why was the score 412? What factors drove that outcome? What did the assessment consider in respect of this consumer’s specific income, obligations, and financial position?

A black-box model cannot answer those questions. A model with documented factor contributions, reason codes, and a plain-language narrative tied to verified financial data can. The difference between those two outcomes, in an NCT hearing, an NCR inspection, or a reckless credit challenge, is the difference between a defensible position and an exposed one.

For credit providers running automated systems, the risk compounds with scale. The NCA risks of full automation extend beyond individual application outcomes. A systematic pattern of unexplainable declines, or approvals, is the kind of finding that triggers enforcement action rather than individual remediation.

What explainability actually looks like in practice

Explainability in credit risk models does not require that the model be simple. Complex models can be made interpretable. The question is whether the output includes factor-level contribution information that allows the decision to be understood in human terms.

Two technical approaches are widely used in the industry for model interpretability:

LIME (Local Interpretable Model-agnostic Explanations) creates a simplified local approximation of the model’s behaviour around a specific prediction. For a given application, LIME identifies which input features most influenced the output and by approximately how much. This allows the model’s behaviour on a specific case to be explained even if the global model is complex.

SHAP (SHapley Additive exPlanations) uses game theory concepts to calculate each feature’s marginal contribution to a prediction across all possible feature orderings. SHAP values are theoretically well-grounded and produce consistent, additive feature importance scores that sum to the model output. For a credit decision, SHAP analysis can produce a statement like: “The income stability factor contributed −42 points to this score; the gambling spend factor contributed −28 points; the debt-service ratio contributed −31 points.”

Beyond technical interpretability tools, explainability in a credit context requires translation into plain-language narrative. A SHAP value of −42 for the income stability factor is not accessible to most credit officers or consumers. A plain-language translation is: “Monthly income varied by more than 35% across the review period. This level of income variability increases the risk that a fixed instalment cannot be reliably serviced.” That is an explanation. That is what Section 81 is looking for.

The practical components of an explainable credit decision record:

This is not a description of the model’s internal workings. It is a record of what the model found in this consumer’s specific data and how those findings contributed to the assessment outcome. The credit officer’s review of this record constitutes the human engagement with the substantive assessment that Section 81 requires.

The human-in-the-loop requirement

The Section 81 assessment obligation rests on the credit provider as an organisation, not on its systems. Systems are tools. The obligation is the credit provider’s. This means that a fully automated decision (one where no human reviews the decision before it is communicated to the consumer) is structurally at odds with the NCA’s substantive assessment requirement, regardless of how sophisticated the model is.

This does not mean manual underwriting for every application. It means that the human decision-maker in the workflow, however briefly they engage with any individual application, must be engaging with substantive reasoning, not just approving or declining based on a score. The system provides structured, explainable reasoning. The human reviews that reasoning and makes the decision. The audit trail documents both the system’s output and the human’s engagement with it.

In practice, this typically means: the system scores and produces a decision recommendation with supporting reasoning; applications in the “Clear” band above a defined confidence threshold are approved with logged human review; applications in “Caution” or “Review” bands require active human decision with documented reasoning. The system handles scale. The human handles substantive judgment at the thresholds where it matters most.

For the behavioural credit scoring framework, this human-in-the-loop design is built into the product philosophy. AffyScore does not decline. It produces a structured recommendation (Clear, Caution, or Review) with sub-factor reason codes, a plain-language narrative, and an affordability calculation. The credit officer receives structured reasoning, not a raw score. The decision they make is informed by the system’s analysis but it is their decision, documented in the audit file alongside AffyScore’s output. That structure satisfies Section 81’s substantive requirement while maintaining the operational efficiency that automated analysis provides.

The audit trail requirement

Beyond the explainability requirement, credit providers need to be able to retrieve the assessment record for a specific application 12 months later, or whenever an NCR inspection or NCT challenge requires it. This is a record retention obligation, and it applies to automated system outputs exactly as it applies to manual assessment records.

For automated systems, this means that the decision output, including the reasoning that supported it, must be stored in a retrievable format linked to the specific application. A log file that records “application ID 44382: score 412, decline” does not satisfy this requirement. A stored decision pack that contains the full assessment record described above (income, obligations, expense norms, factor contributions, recommendation narrative) does.

The NCR v Tsoelli 2023 cancellation case arose in part because the credit provider could not produce assessment records for sampled files from their application history. The inability to retrieve records (whether because they were never created, were deleted, or were stored in a format that cannot be extracted) is a compliance failure independent of whether the underlying assessments were substantively sound. Build the record. Store it retrievably. Test the retrieval before the auditors test it for you.

Frequently asked questions

What does NCA Section 81 require when a credit application is declined?

Section 81 requires credit providers to conduct a substantive affordability assessment before entering into or declining a credit agreement. Where the application is declined on affordability grounds, the credit provider must be able to articulate the basis for that decision in terms of the consumer’s verified income, obligations, and expenses. “The model scored you below our threshold” describes a process but does not explain the substantive assessment that the NCA requires.

What did the March 2026 court judgment find about automated credit decisioning?

The March 2026 court found that Section 81 requires genuine substantive assessment, not procedural compliance. Going through the motions (pulling a score, comparing it to a threshold, outputting a decision) does not satisfy the NCA’s assessment obligation if the credit provider cannot explain the reasoning behind the decision in substantive terms.

Does POPIA require credit providers to explain automated decisions?

South Africa does not yet have a GDPR-equivalent statutory right to algorithmic explanation. However, POPIA’s conditions for lawful processing (transparency, purpose limitation, and accountability) create a directional obligation toward disclosure of automated processing logic. The Information Regulator’s April 2025 amendments strengthened consumer rights to object to automated processing.

What is the difference between a risk score output and an assessment record?

A risk score is a numerical output from a model. An assessment record is a documented explanation of the factors that drove the assessment outcome: income level and source, existing obligations enumerated, expense norms applied, red flags identified, and the resulting discretionary income calculation. The NCA’s substantive assessment requirement is satisfied by the record, not the score.

Is full credit decision automation compliant with the NCA?

Full automation (where a system produces a decision with no human involvement and no documented reasoning) is a compliance risk under Section 81’s substantive assessment requirement. The NCA’s assessment obligation rests on the credit provider as an organisation, not on its systems. Human oversight of the system’s output, supported by an explainable decision record, is the risk-managed approach.

This article is general information for credit providers and does not constitute professional legal or financial advice. Specific regulatory requirements may vary. Always verify against current NCA legislation, the March 2026 judgment, and NCR guidelines before acting.

Automate your bank statement extraction and affordability preparation

Book a demo to see how AffyScore works with your lending workflow.

Book a demo