Support Vector Machines (SVM) parameters

Overview:

Support Vector Machines (SVM) are powerful classifiers for various types of data. OtasML offers a comprehensive set of parameters for configuring SVMs to achieve optimal performance. Below is a detailed guide to the configurable parameters for SVMs in OtasML.

Configurations page:

The Configurations page allows users to adjust various parameters of the SVM. Here are the details:

C

• Default Value: 1.0
• Description: The regularization parameter, C, controls the trade-off between achieving a low training error and a low testing error.
  • Regularization: A smaller C value leads to a wider margin with some misclassifications (soft margin). A larger C minimizes misclassifications, leading to a narrower margin (hard margin).
  • Overfitting vs. Underfitting: A very small C can result in underfitting, while a very large C can result in overfitting.
  • Bias-Variance Trade-off: A smaller C encourages higher bias and lower variance; a larger C results in lower bias and potentially higher variance.
• Warning: Must be strictly positive.

Kernel

• Default Value: rbf
• Description: The kernel is a crucial component that defines the type of transformation applied to the input data to map it into a higher-dimensional space. The kernel trick allows SVMs to create non-linear decision boundaries in this transformed space without explicitly computing the coordinates of the data in that space. The choice of kernel determines how the SVM separates classes that are not linearly separable in the original feature space. SVMs can work with various types of kernels to handle different types of data distributions and classification problems.
  • Linear Kernel (linear): Represents the standard dot product between input features. It's suitable for linearly separable data.
  • Polynomial Kernel (poly): This kernel uses the polynomial function to transform the data into a higher-dimensional space. The degree of the polynomial can be controlled with the degree parameter.
  • Radial Basis Function (RBF) Kernel (rbf): Also known as the Gaussian kernel, it maps data into an infinite-dimensional space using a radial basis function. The gamma parameter controls the width of the Gaussian distribution.
  • Sigmoid Kernel (sigmoid): This kernel is based on the hyperbolic tangent function and can be useful for data that exhibits sigmoid-like behavior.

Degree

• Default Value: 3
• Description: The degree parameter refers to the degree of the polynomial kernel function when the kernel type is set to poly. The degree parameter allows you to specify the degree of the polynomial in the kernel function. Higher polynomial degrees lead to more complex transformations and decision boundaries in the higher-dimensional space.

Gamma

• Default Value: scale
• Description: Controls the influence of individual data points on the decision boundary. It shapes the decision regions.
  • Warning: Choose appropriately to avoid overfitting or underfitting.

Coef0

• Default Value: 0.0
• Description: The coef0 parameter (also denoted as r) is an independent term added to the dot product of the input feature vectors in the kernel function. The coef0 parameter is used in kernels like the polynomial kernel poly and the sigmoid kernel sigmoid.

Shrinking

• Default Value: True
• Description: It refers to a technique used to improve the efficiency of the SVM training process and reduce the computational resources required while maintaining the accuracy of the model. The shrinking technique works by identifying support vectors during the optimization process that are far from the margin or have little impact on the final solution. Support vectors are the data points that lie closest to the decision boundary (hyperplane) and contribute most to defining the boundary.

Probability

• Default Value: True
• Description: Enables or disables probability estimates for predicted classes.

Tol

• Default Value: 1e-3
• Description: Tolerance for convergence during training. Controls when the optimization algorithm should stop iterating.
• Warning: A smaller tol value requires the algorithm to achieve higher precision, potentially leading to longer training times but a more accurate solution. A larger tol value allows for quicker convergence but may result in a slightly less accurate solution.

Cache Size

• Default Value: 200
• Description: The cache size parameter refers to the amount of memory (in megabytes) allocated for caching intermediate results during the training process. The optimization problem's solution involves working with matrices and calculations that can be memory-intensive. The cache size parameter allows you to control the amount of memory allocated for caching intermediate results, which can help speed up the training process and reduce memory usage.

Class Weight

• Default Value: None
• Description: The class weight parameter is used to assign different weights to classes to address class imbalance issues. Class imbalance occurs when one class has significantly fewer samples than another class in the training data. In such cases, SVMs might not perform well because they can become biased towards the majority class.
  • Warning: Set the parameter C to class weight * C. If not given, all classes are supposed to have weight one. The balanced mode uses the values of output to automatically adjust weights inversely proportional to class frequencies in the input data.

Verbose

• Default Value: False
• Description: The verbose parameter allows you to control the amount of information or output that is displayed during the training process. It helps you monitor the progress of the training algorithm and can be particularly useful for understanding how long the training will take and whether any issues or errors occur.
• Warning: Note that this setting takes advantage of a per-process runtime setting in libsvm that, if enabled, may not work properly in a multithreaded context.

Max Iter

• Default Value: -1
• Description: It sets a hard limit on the number of iterations that the optimization solver can perform. If the solver reaches this maximum number of iterations before convergence, the training process will stop, even if convergence has not been achieved.
• Warning: Hard limit on iterations within solver, or -1 for no limit.

Decision Function Shape

• Default Value: ovr
• Description: It allows you to specify the shape of the decision function output when the SVM is used for multi-class classification problems.
  • ovr (One-Versus-Rest): In this mode, for each class, the classifier is trained to distinguish that class from all other classes combined.
  • ovo (One-Versus-One): In this mode, a binary classifier is trained for each pair of classes (i.e., Class 0 vs. Class 1, Class 0 vs. Class 2, Class 1 vs. Class 2, etc.).

Break Ties

• Default Value: False
• Description: If true, Decision function shape = ovr, and number of classes > 2, predict will break ties according to the confidence values of decision_function, otherwise the first class among the tied classes is returned.
• Warning: Please note that breaking ties comes at a relatively high computational cost compared to a simple predict.

Random State

• Default Value: None
• Description: Controls the pseudo random number generation for shuffling the data for probability estimates. Ignored when probability is False. Pass an int for reproducible output across multiple function calls.

Test Size

• Default Value: 0.2
• Description: The test size parameter is used when splitting the dataset into these subsets, and it specifies the portion of the data that will be used for testing.

Train Size

• Default Value: 0.8
• Description: The train size parameter is used when splitting the dataset into these subsets, and it specifies the portion of the data that will be used for model training.

Conclusion

OtasML’s configuration options for SVMs empower users to build robust models tailored to their specific data and classification problems. Adjusting parameters like C, kernel, gamma, and class weight can significantly impact the model's performance. Explore these settings to fine-tune your SVM and achieve superior predictive accuracy with OtasML.