AKS CSI NFS 4.1 Mount: stunnel encryption reports certificate not trusted (unable to get local issuer certificate) on Azure China cloud

[shanshanyan_microsoft]

Environment Information:

• AKS Version: 1.33.3
• Node OS: Ubuntu Linux
• CSI Driver Version: v1.33.5

Problem Description:
When using AzureFile CSI driver for NFS 4.1 mount with encryption (encryptInTransit) enabled, stunnel is used for encrypted connections. However, certificate trust errors are reported:

Oct 21 08:16:20 aks-system-26797224-vmss00000D stunnel: LOG4[27153]: CERT: Pre-verification error: unable to get local issuer certificate
Oct 21 08:16:20 aks-system-26797224-vmss00000D stunnel: LOG4[27153]: Rejected by CERT at depth=1: C=US, O=DigiCert Inc, CN=DigiCert Basic RSA CN CA G2
Oct 21 08:16:20 aks-system-26797224-vmss00000D stunnel: LOG3[27153]: SSL_connect: ../ssl/statem/statem_clnt.c:1883: error:0A000086:SSL routines::certificate verify failed

However, when verifying the certificate chain using openssl s_client on the AKS node, it is trusted:

CONNECTED(00000003)
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, CN = DigiCert Basic RSA CN CA G2
verify return:1
depth=0 C = CN, ST = Shanghai, O = "Shanghai Blue Cloud Technology Co., Ltd.", CN = *.file.core.chinacloudapi.cn
verify return:1

Root and intermediate certificates have been confirmed as trusted.

Reproduction Steps:

1. Configure NFS mount per the official README, with StorageClass parameter set to encryptInTransit: "true".
2. On the AKS node, openssl can successfully verify the certificate of the Azure NFS service.
3. CSI stunnel mount NFS 4.1 reports the above error.

Expected Behavior:
stunnel should correctly trust the Azure NFS service certificate, and the mount should succeed.

Related Documentation and Code:

• NFS 4.1 Support & Encryption Configuration
• Driver Parameter Documentation
• NFS Mount Code Implementation (NodeStageVolume)

  // Relevant code handling encryptInTransit
  mountFsType = nfs
  if newOptions, exists := removeOptionIfExists(mountOptions, encryptInTransitField); exists {
    klog.V(2).Infof("encryptInTransit is set in mountOptions(%v), enabling encryptInTransit", mountOptions)
    encryptInTransit = true
    mountOptions = newOptions
  }
  if encryptInTransit {
    mountFsType = aznfs
  }
  if mountFsType == aznfs && !d.enableAzurefileProxy {
    return nil, status.Error(codes.InvalidArgument, "encryptInTransit is only available when azurefile-proxy is enabled")
  }
  // Actually mount with proxy
  if mountFsType == aznfs {
    klog.V(2).Infof("encryptInTransit is enabled, mount by azurefile-proxy")
    if err := d.mountWithProxy(ctx, source, cifsMountPath, mountFsType, mountOptions, sensitiveMountOptions); err != nil {
      return nil, status.Errorf(codes.Internal, "mount with proxy failed for %s with error: %v", cifsMountPath, err)
    }
    klog.V(2).Infof("mount with proxy succeeded for %s", cifsMountPath)
  }

• stunnel/azurefile-proxy Initialization Script

  HOST_CMD="nsenter --mount=/proc/1/ns/mnt"
  if [ "$ENABLE_MI_AUTH" = "true" ];then
    echo "set up /etc/krb5.conf on host"
    printf '[libdefaults]\ndefault_ccache_name = FILE:/var/lib/kubelet/kerberos/krb5cc_%s\n' "%{uid}" > /host/etc/krb5.conf
    mkdir -p /var/lib/kubelet/kerberos
    echo "set up /etc/azfilesauth/config.yaml on host"
    mkdir -p /host/etc/azfilesauth
  fi

Questions/Requests:

• Why is stunnel unable to trust the Azure NFS service certificate? Are any CA/intermediate certificates missing? How can stunnel configuration be fixed to correctly trust the certificate?
• Is it necessary to manually mount certificates into the CSI driver container?
• Are there recommended stunnel configurations or parameters for this scenario?

Please assist with investigating and fixing this issue. Thank you!

[andyzhangx]

if you run following command to mount aznfs (0.3.15) on the agent node directly, it would fail either:

mkdir /tmp/test
mount -v -t aznfs -o vers=4,minorversion=1,sec=sys accountname.file.core.chinacloudapi.cn:/accountname/sharename /tmp/test

[andyzhangx]

I filed an issue here: Azure/AZNFS-mount#278