predict_proba results are (wildly) inconsistent between sklearn.svm.SVC and sklearnex.svm.SVC

[thwit]

Summary
Inconsistent output from predict_proba on an SVC classifier, when compared with the scikit-learn implementation.

Minimal reproducible example

from sklearn.svm import SVC as SklearnSVC
from sklearnex.svm import SVC as SklearnexSVC

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
import numpy as np

# Generate data
X, y = make_classification(n_samples=15000, n_features=50, n_classes=10, n_informative=20, n_redundant=5, n_clusters_per_class=4, class_sep=0.2, flip_y=0.05, random_state=42)

# Split data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Train sklearn SVC
sklearn_svc = SklearnSVC(probability=True, random_state=42)
sklearn_svc.fit(X_train, y_train)
sklearn_proba = sklearn_svc.predict_proba(X_test)
sklearn_proba_max = sklearn_proba.max(1)

# Train sklearnex SVC
sklearnex_svc = SklearnexSVC(probability=True, random_state=42)
sklearnex_svc.fit(X_train, y_train)
sklearnex_proba = sklearnex_svc.predict_proba(X_test)
sklearnex_proba_max = sklearnex_proba.max(1)

# Visualize maximal probabilites per sample
fig, ax = plt.subplots()


# Get the index of the maximum value for each row in both arrays
predicted_class_sklearn = np.argmax(sklearn_proba, axis=1)
predicted_class_sklearnex = np.argmax(sklearnex_proba, axis=1)
mask_same_class_predicted = predicted_class_sklearnex == predicted_class_sklearn

# Scatter point is red if they predicted different classes, black if they agree on predicted class
c = ["black" if agreement else "red" for agreement in mask_same_class_predicted]
ax.scatter(sklearn_proba_max, sklearnex_proba_max, s=5, c=c, alpha=.5)

# Add legend
from matplotlib.lines import Line2D
legend_elements = [
    Line2D([0], [0], marker='o', color='w', markerfacecolor='black', markersize=8, alpha=0.5, label='Same class predicted'),
    Line2D([0], [0], marker='o', color='w', markerfacecolor='red', markersize=8, alpha=0.5, label='Different class predicted')
]
ax.legend(handles=legend_elements, loc='best', frameon=False)

ax.spines[["right", "top"]].set_visible(False)
ax.set_title("sklearn vs sklearnex max probability per sample")
ax.set_xlabel("sklearn max probability")
ax.set_ylabel("sklearnex max probability")
ax.set_xlim([0,1])
ax.set_ylim([0,1])

Expected outcome
Output from sklearnex.svm.predict_proba should be similar/consistent with output from sklearn.svm.predict_proba. This would produce a plot with scatter points lying on a roughly straight line going from (0,0) to (1,1).

Actual outcome
Output from sklearnex.svm.predict_proba is very different from the output of sklearn.svm.predict_proba. The former almost always predicts high scores, while the latters predictions are more evenly distributed from 0-1. See plots below:

Environment
Red Hat Enterprise Linux 8.10 (Ootpa)
scikit-learn version: 1.7.2
scikit-learn-intelex version: 2199.9.9
numpy version: 2.3.4
Python version: 3.11.13

Wrap up
This is my first issue on this project. Please let me know, if I need to provide additional information.

[david-cortes-intel]

Thanks for the bug report.

I can confirm that the problem is reproducible.

Unfortunately, this kind of variation is somewhat expectable from the algorithm used to compute probabilities, the more so since LibSVM (used by scikit-learn's SVC) has some unexpected quirks in its implementation of this algorithm:
scikit-learn/scikit-learn#25631 (comment)
scikit-learn/scikit-learn#13662

That, plus the maintainers of scikit-learn have expressed some desire to deprecate this parameter probability=True due to these kinds of differences:
scikit-learn/scikit-learn#31222 (comment)

Nevertheless, if you look at class-by-class probabilities, you might find that they are not too different relative to each other.

Example:

import polars as pl
import plotnine as p9
df_probs = pl.DataFrame({
    "sklearn p0": sklearn_proba[:, 0],
    "sklearnex p0": sklearnex_proba[:, 0],
    "sklearn p1": sklearn_proba[:, 1],
    "sklearnex p1": sklearnex_proba[:, 1],
})
(
    p9.ggplot(df_probs, p9.aes(x="sklearnex p0", y="sklearn p0"))
    + p9.geom_point()
    + p9.geom_abline(intercept=0, slope=1, color="red")
).show()

What's more, you'll find that they are a lot closer to what you'd get from using scikit-learn's CalibratedClassifierCV, which scikit-learn-intelex is using internally:

import numpy as np
from sklearnex.svm import SVC
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.calibration import CalibratedClassifierCV
from sklearn.model_selection import StratifiedKFold

X, y = make_classification(
    n_samples=15000,
    n_features=50,
    n_classes=10,
    n_informative=20,
    n_redundant=5,
    n_clusters_per_class=4,
    class_sep=0.2,
    flip_y=0.05,
    random_state=42,
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

svc = SVC(probability=True, random_state=42,)
svc.fit(X_train, y_train)
sklearnex_proba = svc.predict_proba(X_test)

calib = CalibratedClassifierCV(
    estimator=svc,
    ensemble=False,
    cv=StratifiedKFold(),
    n_jobs=-1,
).fit(X_train, y_train)

calib_proba = calib.predict_proba(X_test)

np.corrcoef(
    sklearnex_proba[:, 0],
    calib_proba[:, 0],
)

array([[1.       , 0.9663656],
       [0.9663656, 1.       ]])

[thwit]

Hi David

Thanks for the quick and thorough response! I'll read a bit into the other currently known issues you mention along with the usage of CalibratedClassifierCV.