Software Testing Trends LogoSoftware Testing Trends

Giskard AI - Comprehensive Testing Platform

Complete guide to Giskard AI, the open-source Python library for comprehensive ML model testing, including vulnerability detection, bias assessment, and performance monitoring.

Giskard AI: Comprehensive ML Model Testing

Giskard AI is a revolutionary open-source Python library designed specifically for testing and monitoring Machine Learning models. It provides an extensive suite of testing capabilities to detect vulnerabilities, assess performance, and ensure model reliability across diverse scenarios and data distributions.

What is Giskard AI?

Overview

Giskard AI is a comprehensive testing framework that automatically scans ML models to detect:

  • Performance Issues: Accuracy degradation and prediction failures
  • Data Issues: Distribution drift, outliers, and quality problems
  • Bias and Fairness: Discrimination across different groups
  • Robustness: Vulnerability to adversarial attacks
  • Ethical Concerns: Harmful outputs and inappropriate behavior

Key Features

🔍 Automatic Vulnerability Scanning

  • Automated detection of over 30 types of ML vulnerabilities
  • Comprehensive model testing without manual test case creation
  • Real-time vulnerability scoring and prioritization

📊 Interactive Model Debugging

  • Visual debugging tools for understanding model behavior
  • Interactive data exploration and analysis capabilities
  • Detailed vulnerability reports with actionable insights

🔄 Continuous Monitoring

  • Production model monitoring and alerting
  • Performance drift detection over time
  • Automated retraining recommendations

🎯 Domain-Specific Testing

  • Specialized testing for NLP, computer vision, and tabular models
  • Industry-specific test suites (finance, healthcare, retail)
  • Custom test development framework

Installation and Setup

Quick Installation

# Install Giskard AI
pip install giskard

# For development with all dependencies
pip install "giskard[all]"

# For specific model types
pip install "giskard[nlp]"      # NLP models
pip install "giskard[vision]"   # Computer vision models
pip install "giskard[llm]"      # Large Language Models

Environment Setup

import giskard
import pandas as pd
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

# Set up logging for detailed output
import logging
logging.basicConfig(level=logging.INFO)

print(f"Giskard version: {giskard.__version__}")

Quick Start Example

# Load your data and model
df = pd.read_csv('your_dataset.csv')
X = df.drop('target', axis=1)
y = df['target']

# Train a simple model
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
model = RandomForestClassifier()
model.fit(X_train, y_train)

# Wrap model and dataset for Giskard
giskard_model = giskard.Model(
    model=model,
    model_type="classification",
    feature_names=X.columns.tolist(),
    name="My ML Model"
)

giskard_dataset = giskard.Dataset(
    df=X_test,
    target=y_test,
    name="Test Dataset"
)

# Run comprehensive scan
scan_results = giskard.scan(giskard_model, giskard_dataset)
print(scan_results)

Core Capabilities

1. Model Wrapping and Integration

Giskard supports a wide range of ML frameworks:

Scikit-learn Models

from sklearn.ensemble import RandomForestClassifier
import giskard

# Train your model
model = RandomForestClassifier()
model.fit(X_train, y_train)

# Wrap for Giskard
giskard_model = giskard.Model(
    model=model,
    model_type="classification",  # or "regression"
    feature_names=feature_names,
    name="Random Forest Classifier"
)

Deep Learning Models (TensorFlow/Keras)

import tensorflow as tf
import giskard

# Load your trained model
model = tf.keras.models.load_model('my_model.h5')

# Create prediction function
def predict_fn(df):
    return model.predict(df.values)

# Wrap for Giskard
giskard_model = giskard.Model(
    model=predict_fn,
    model_type="classification",
    feature_names=feature_names,
    name="Deep Learning Model"
)

PyTorch Models

import torch
import giskard

# Load your trained model
model = torch.load('model.pth')
model.eval()

def predict_fn(df):
    with torch.no_grad():
        tensor_input = torch.tensor(df.values, dtype=torch.float32)
        predictions = model(tensor_input)
        return predictions.numpy()

giskard_model = giskard.Model(
    model=predict_fn,
    model_type="classification",
    feature_names=feature_names,
    name="PyTorch Model"
)

Hugging Face Transformers

from transformers import pipeline
import giskard

# Load pre-trained model
classifier = pipeline("text-classification",
                     model="bert-base-uncased")

def predict_fn(df):
    texts = df['text'].tolist()
    results = classifier(texts)
    return [r['label'] for r in results]

giskard_model = giskard.Model(
    model=predict_fn,
    model_type="classification",
    feature_names=['text'],
    name="BERT Classifier"
)

2. Dataset Preparation

Tabular Data

# Basic dataset creation
giskard_dataset = giskard.Dataset(
    df=test_df,
    target=y_test,
    name="Test Dataset",
    target_name="target_column"
)

# With additional metadata
giskard_dataset = giskard.Dataset(
    df=test_df,
    target=y_test,
    name="Customer Churn Dataset",
    target_name="churn",
    cat_columns=['gender', 'contract_type'],  # Categorical columns
    num_columns=['age', 'monthly_charges']    # Numerical columns
)

Text Data

# NLP dataset
giskard_dataset = giskard.Dataset(
    df=text_df,
    target=labels,
    name="Sentiment Analysis Dataset",
    target_name="sentiment",
    column_types={'text': 'text'}  # Specify text columns
)

Image Data

# Computer vision dataset
giskard_dataset = giskard.Dataset(
    df=image_df,
    target=labels,
    name="Image Classification Dataset",
    target_name="class",
    column_types={'image': 'image'}  # Specify image columns
)

3. Comprehensive Model Scanning

Basic Scan

# Run automatic vulnerability scan
scan_results = giskard.scan(giskard_model, giskard_dataset)

# Display results summary
print(f"Found {len(scan_results.issues)} issues")
print(scan_results)

# Get detailed issue information
for issue in scan_results.issues:
    print(f"Issue: {issue.issue_type}")
    print(f"Severity: {issue.level}")
    print(f"Description: {issue.description}")
    print("---")

Advanced Scan Configuration

# Customize scan parameters
scan_results = giskard.scan(
    giskard_model,
    giskard_dataset,
    only=[
        "performance",
        "bias",
        "data_leakage",
        "robustness"
    ],  # Specific vulnerability types
    max_issues_per_detector=5,  # Limit issues per detector
    threshold=0.5  # Severity threshold
)

Scan for Specific Vulnerabilities

# Performance issues
performance_scan = giskard.scan(
    giskard_model,
    giskard_dataset,
    only=["performance"]
)

# Bias and fairness
bias_scan = giskard.scan(
    giskard_model,
    giskard_dataset,
    only=["bias", "ethics"]
)

# Robustness testing
robustness_scan = giskard.scan(
    giskard_model,
    giskard_dataset,
    only=["robustness", "adversarial"]
)

4. Issue Types and Detection

Performance Issues

# Issues detected:
# - Low accuracy on specific data slices
# - Inconsistent predictions
# - Poor confidence calibration
# - Regression in performance metrics

# Example: Performance on data slice
issue_example = {
    'type': 'underperformance',
    'slice': "age > 65 AND income < 30000",
    'metric': 'accuracy',
    'value': 0.62,
    'threshold': 0.80,
    'description': 'Model underperforms on elderly low-income segment'
}

Bias and Fairness Issues

# Issues detected:
# - Demographic parity violations
# - Equalized odds disparities
# - Individual fairness problems
# - Proxy discrimination

# Example: Bias detection
bias_issue = {
    'type': 'demographic_parity',
    'protected_attribute': 'gender',
    'privileged_group': 'male',
    'unprivileged_group': 'female',
    'parity_difference': 0.15,
    'threshold': 0.05,
    'description': 'Significant disparity in positive predictions between genders'
}

Data Quality Issues

# Issues detected:
# - Distribution drift
# - Outliers and anomalies
# - Missing value patterns
# - Data leakage indicators

# Example: Data drift
drift_issue = {
    'type': 'data_drift',
    'feature': 'income',
    'drift_score': 0.75,
    'threshold': 0.50,
    'description': 'Significant distribution drift in income feature'
}

Robustness Issues

# Issues detected:
# - Adversarial vulnerability
# - Input perturbation sensitivity
# - Edge case failures
# - Overconfidence problems

# Example: Adversarial vulnerability
robustness_issue = {
    'type': 'adversarial_vulnerability',
    'attack_method': 'gradient_based',
    'success_rate': 0.85,
    'perturbation_budget': 0.1,
    'description': 'Model vulnerable to gradient-based adversarial attacks'
}

5. Interactive Debugging and Analysis

Model Inspector

# Launch interactive model inspector
giskard.inspector(giskard_model, giskard_dataset)
# Opens web interface for model exploration

# Programmatic inspection
inspector = giskard.Inspector(giskard_model, giskard_dataset)

# Analyze feature importance
feature_importance = inspector.explain_global()
print(feature_importance)

# Individual prediction explanation
explanation = inspector.explain_local(sample_index=0)
print(explanation)

Data Visualization

# Visualize data distribution
giskard.visualize.data_distribution(giskard_dataset)

# Performance visualization
giskard.visualize.performance_analysis(
    giskard_model,
    giskard_dataset
)

# Bias visualization
giskard.visualize.bias_analysis(
    giskard_model,
    giskard_dataset,
    protected_columns=['gender', 'age_group']
)

Custom Analysis

# Create custom analysis
def analyze_high_value_customers(model, dataset):
    """Analyze model performance on high-value customers."""
    high_value_mask = dataset.df['customer_value'] > 1000
    high_value_data = dataset.df[high_value_mask]

    predictions = model.predict(high_value_data)
    accuracy = calculate_accuracy(predictions, dataset.target[high_value_mask])

    return {
        'segment': 'high_value_customers',
        'size': len(high_value_data),
        'accuracy': accuracy,
        'issues': identify_issues(predictions, dataset.target[high_value_mask])
    }

# Run custom analysis
analysis_result = analyze_high_value_customers(giskard_model, giskard_dataset)
print(analysis_result)

Domain-Specific Applications

Natural Language Processing

Text Classification Testing

import giskard
from transformers import pipeline

# Load pre-trained sentiment analysis model
sentiment_pipeline = pipeline("text-classification",
                             model="cardiffnlp/twitter-roberta-base-sentiment-latest")

def predict_sentiment(df):
    texts = df['text'].tolist()
    results = sentiment_pipeline(texts)
    return [r['label'] for r in results]

# Wrap model
giskard_model = giskard.Model(
    model=predict_sentiment,
    model_type="classification",
    feature_names=['text'],
    name="Sentiment Analysis Model"
)

# Create text dataset
text_data = pd.DataFrame({
    'text': ["I love this product!", "This is terrible", "Okay product"],
    'sentiment': ['POSITIVE', 'NEGATIVE', 'NEUTRAL']
})

giskard_dataset = giskard.Dataset(
    df=text_data,
    target=text_data['sentiment'],
    name="Sentiment Dataset",
    column_types={'text': 'text'}
)

# NLP-specific scan
nlp_scan = giskard.scan(
    giskard_model,
    giskard_dataset,
    only=[
        "text_perturbation",      # Typo robustness
        "text_length_sensitivity", # Length variations
        "language_detection",      # Mixed languages
        "hate_speech",            # Harmful content
        "bias_language"           # Linguistic bias
    ]
)

print(nlp_scan)

Advanced NLP Testing

# Custom NLP vulnerability tests
def test_punctuation_robustness(model, dataset):
    """Test model robustness to punctuation changes."""
    original_texts = dataset.df['text']

    # Remove all punctuation
    no_punct_texts = original_texts.str.replace(r'[^\w\s]', '', regex=True)

    original_preds = model.predict(dataset.df)
    modified_preds = model.predict(pd.DataFrame({'text': no_punct_texts}))

    # Calculate prediction consistency
    consistency = (original_preds == modified_preds).mean()

    return {
        'test_name': 'punctuation_robustness',
        'consistency_score': consistency,
        'pass_threshold': 0.80,
        'status': 'PASS' if consistency >= 0.80 else 'FAIL'
    }

# Run custom NLP test
punct_test = test_punctuation_robustness(giskard_model, giskard_dataset)
print(punct_test)

Computer Vision

Image Classification Testing

import torch
import torchvision.transforms as transforms
from PIL import Image

# Load pre-trained model
model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet18', pretrained=True)
model.eval()

def predict_image(df):
    """Predict image classes."""
    transform = transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406],
                           std=[0.229, 0.224, 0.225]),
    ])

    predictions = []
    for img_path in df['image_path']:
        image = Image.open(img_path)
        input_tensor = transform(image).unsqueeze(0)

        with torch.no_grad():
            output = model(input_tensor)
            pred = output.argmax(dim=1).item()
            predictions.append(pred)

    return predictions

# Wrap computer vision model
giskard_model = giskard.Model(
    model=predict_image,
    model_type="classification",
    feature_names=['image_path'],
    name="Image Classifier"
)

# Computer vision specific tests
cv_scan = giskard.scan(
    giskard_model,
    giskard_dataset,
    only=[
        "image_brightness",      # Brightness variations
        "image_contrast",        # Contrast changes
        "image_blur",           # Blur effects
        "image_noise",          # Noise addition
        "image_rotation",       # Rotation robustness
        "adversarial_images"    # Adversarial examples
    ]
)

Tabular Data Models

Financial Risk Model Testing

# Credit scoring model example
credit_model = RandomForestClassifier()
credit_model.fit(X_train, y_train)

giskard_model = giskard.Model(
    model=credit_model,
    model_type="classification",
    feature_names=feature_names,
    name="Credit Risk Model"
)

# Financial-specific testing
financial_scan = giskard.scan(
    giskard_model,
    giskard_dataset,
    only=[
        "bias",                  # Fair lending compliance
        "performance",           # Risk model accuracy
        "robustness",           # Economic stability
        "data_leakage",         # Information leakage
        "feature_importance",    # Explainability
        "ethical_ai"            # Regulatory compliance
    ]
)

# Custom financial tests
def test_fair_lending_compliance(model, dataset):
    """Test compliance with fair lending regulations."""
    protected_attributes = ['race', 'gender', 'age_group']

    results = {}
    for attr in protected_attributes:
        if attr in dataset.df.columns:
            # Calculate approval rates by group
            groups = dataset.df[attr].unique()
            approval_rates = {}

            for group in groups:
                mask = dataset.df[attr] == group
                group_data = dataset.df[mask]
                predictions = model.predict(group_data)
                approval_rate = (predictions == 1).mean()
                approval_rates[group] = approval_rate

            # Calculate disparate impact ratio
            max_rate = max(approval_rates.values())
            min_rate = min(approval_rates.values())
            impact_ratio = min_rate / max_rate if max_rate > 0 else 0

            results[attr] = {
                'approval_rates': approval_rates,
                'disparate_impact_ratio': impact_ratio,
                'compliant': impact_ratio >= 0.80  # 80% rule
            }

    return results

compliance_test = test_fair_lending_compliance(giskard_model, giskard_dataset)
print(compliance_test)

Production Monitoring

Continuous Monitoring Setup

# Set up production monitoring
monitor = giskard.Monitor(
    model=giskard_model,
    reference_dataset=giskard_dataset,
    monitoring_config={
        'drift_threshold': 0.1,
        'performance_threshold': 0.05,
        'bias_threshold': 0.02,
        'check_frequency': 'daily'
    }
)

# Add production data for monitoring
production_data = load_production_data()  # Your data loading function
monitor.add_batch(production_data)

# Check for issues
monitoring_results = monitor.check_issues()
print(monitoring_results)

Alerting and Notifications

# Configure alerts
alert_config = {
    'email_recipients': ['ml-team@company.com'],
    'slack_webhook': 'https://hooks.slack.com/...',
    'severity_threshold': 'medium',
    'alert_frequency': 'immediate'
}

monitor.configure_alerts(alert_config)

# Custom alert conditions
def custom_alert_condition(results):
    """Define custom conditions for triggering alerts."""
    critical_issues = [
        issue for issue in results.issues
        if issue.level == 'high' and issue.issue_type in ['bias', 'performance']
    ]
    return len(critical_issues) > 0

monitor.add_custom_alert(custom_alert_condition)

Automated Remediation

# Set up automated responses to issues
def automated_response(issue):
    """Automated response to detected issues."""
    if issue.issue_type == 'data_drift':
        # Trigger model retraining pipeline
        trigger_retraining_pipeline()

    elif issue.issue_type == 'bias':
        # Alert compliance team
        send_compliance_alert(issue)

    elif issue.issue_type == 'performance':
        # Scale down model usage
        reduce_model_traffic(percentage=50)

monitor.set_automated_response(automated_response)

Integration with MLOps Pipelines

CI/CD Integration

# GitHub Actions example
"""
name: ML Model Quality Check
on: [push, pull_request]

jobs:
  quality-check:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Set up Python
      uses: actions/setup-python@v2
      with:
        python-version: '3.8'
    - name: Install dependencies
      run: |
        pip install giskard scikit-learn pandas
    - name: Run Giskard scan
      run: |
        python -c "
        import giskard
        from joblib import load
        import pandas as pd

        # Load model and test data
        model = load('model.joblib')
        test_data = pd.read_csv('test_data.csv')

        # Create Giskard objects
        giskard_model = giskard.Model(model=model, model_type='classification')
        giskard_dataset = giskard.Dataset(df=test_data)

        # Run scan
        results = giskard.scan(giskard_model, giskard_dataset)

        # Fail if critical issues found
        critical_issues = [i for i in results.issues if i.level == 'high']
        if critical_issues:
            print(f'Found {len(critical_issues)} critical issues')
            exit(1)
        else:
            print('Quality check passed')
        "
"""

MLflow Integration

import mlflow
import giskard

# Log Giskard results to MLflow
def log_giskard_results(model, dataset, run_name):
    with mlflow.start_run(run_name=run_name):
        # Run Giskard scan
        scan_results = giskard.scan(model, dataset)

        # Log metrics
        mlflow.log_metric("total_issues", len(scan_results.issues))
        mlflow.log_metric("high_severity_issues",
                         len([i for i in scan_results.issues if i.level == 'high']))
        mlflow.log_metric("medium_severity_issues",
                         len([i for i in scan_results.issues if i.level == 'medium']))

        # Log artifacts
        scan_results.to_html("giskard_report.html")
        mlflow.log_artifact("giskard_report.html")

        # Log issue details as parameters
        for i, issue in enumerate(scan_results.issues[:5]):  # Top 5 issues
            mlflow.log_param(f"issue_{i}_type", issue.issue_type)
            mlflow.log_param(f"issue_{i}_level", issue.level)

# Example usage
log_giskard_results(giskard_model, giskard_dataset, "quality_check_v1")

Kubeflow Pipeline Integration

from kfp import dsl
import giskard

@dsl.component
def giskard_quality_check(model_uri: str, data_uri: str) -> str:
    """Kubeflow component for Giskard quality checking."""
    import giskard
    import joblib
    import pandas as pd

    # Load model and data
    model = joblib.load(model_uri)
    data = pd.read_csv(data_uri)

    # Create Giskard objects
    giskard_model = giskard.Model(model=model, model_type="classification")
    giskard_dataset = giskard.Dataset(df=data)

    # Run quality check
    results = giskard.scan(giskard_model, giskard_dataset)

    # Return results summary
    summary = {
        'total_issues': len(results.issues),
        'critical_issues': len([i for i in results.issues if i.level == 'high']),
        'status': 'PASS' if len([i for i in results.issues if i.level == 'high']) == 0 else 'FAIL'
    }

    return str(summary)

@dsl.pipeline(name='ML Pipeline with Quality Gates')
def ml_pipeline_with_quality():
    # Model training step
    train_op = train_model_component()

    # Quality check step
    quality_check_op = giskard_quality_check(
        model_uri=train_op.outputs['model'],
        data_uri="gs://bucket/test_data.csv"
    )

    # Deployment step (conditional on quality check)
    with dsl.Condition(quality_check_op.outputs['status'] == 'PASS'):
        deploy_op = deploy_model_component(train_op.outputs['model'])

Best Practices and Guidelines

Testing Strategy Development

1. Comprehensive Test Planning

# Define testing strategy
testing_strategy = {
    'test_phases': [
        'development_testing',    # During model development
        'pre_deployment_testing', # Before production deployment
        'production_monitoring'   # Continuous monitoring
    ],
    'test_coverage': [
        'performance_testing',    # Accuracy, precision, recall
        'bias_testing',          # Fairness across groups
        'robustness_testing',    # Adversarial and edge cases
        'data_quality_testing',  # Input validation
        'ethical_testing'        # Harmful content detection
    ],
    'quality_gates': {
        'accuracy_threshold': 0.85,
        'bias_threshold': 0.05,
        'robustness_threshold': 0.90
    }
}

2. Test Environment Setup

# Create standardized testing environment
class GiskardTestSuite:
    def __init__(self, model, dataset, config):
        self.model = model
        self.dataset = dataset
        self.config = config
        self.results = {}

    def run_full_suite(self):
        """Run comprehensive test suite."""
        tests = [
            self.test_performance,
            self.test_bias_fairness,
            self.test_robustness,
            self.test_data_quality,
            self.test_ethical_ai
        ]

        for test in tests:
            try:
                result = test()
                self.results[test.__name__] = result
            except Exception as e:
                self.results[test.__name__] = {'error': str(e)}

        return self.generate_report()

    def test_performance(self):
        """Test model performance across different segments."""
        scan_results = giskard.scan(
            self.model, self.dataset,
            only=['performance']
        )
        return self.process_scan_results(scan_results)

    def test_bias_fairness(self):
        """Test for bias and fairness issues."""
        scan_results = giskard.scan(
            self.model, self.dataset,
            only=['bias', 'ethics']
        )
        return self.process_scan_results(scan_results)

    def test_robustness(self):
        """Test model robustness to various perturbations."""
        scan_results = giskard.scan(
            self.model, self.dataset,
            only=['robustness', 'adversarial']
        )
        return self.process_scan_results(scan_results)

    def process_scan_results(self, scan_results):
        """Process and standardize scan results."""
        return {
            'total_issues': len(scan_results.issues),
            'high_severity': len([i for i in scan_results.issues if i.level == 'high']),
            'medium_severity': len([i for i in scan_results.issues if i.level == 'medium']),
            'issues_detail': [
                {
                    'type': issue.issue_type,
                    'level': issue.level,
                    'description': issue.description
                } for issue in scan_results.issues
            ]
        }

# Usage
test_suite = GiskardTestSuite(giskard_model, giskard_dataset, config)
full_results = test_suite.run_full_suite()

3. Custom Test Development

# Create domain-specific custom tests
class CustomFinancialTests:
    """Custom tests for financial ML models."""

    @staticmethod
    def test_economic_stress(model, dataset, stress_scenarios):
        """Test model under economic stress scenarios."""
        results = {}

        for scenario_name, scenario_params in stress_scenarios.items():
            # Apply stress scenario to data
            stressed_data = apply_stress_scenario(dataset.df, scenario_params)
            stressed_dataset = giskard.Dataset(
                df=stressed_data,
                target=dataset.target,
                name=f"Stressed_{scenario_name}"
            )

            # Compare performance
            baseline_accuracy = evaluate_model(model, dataset)
            stressed_accuracy = evaluate_model(model, stressed_dataset)

            results[scenario_name] = {
                'baseline_accuracy': baseline_accuracy,
                'stressed_accuracy': stressed_accuracy,
                'performance_degradation': baseline_accuracy - stressed_accuracy,
                'acceptable': (baseline_accuracy - stressed_accuracy) < 0.10
            }

        return results

    @staticmethod
    def test_regulatory_compliance(model, dataset, regulations):
        """Test compliance with financial regulations."""
        compliance_results = {}

        for reg_name, requirements in regulations.items():
            if reg_name == 'fair_lending':
                result = test_fair_lending_compliance(model, dataset)
            elif reg_name == 'model_explainability':
                result = test_model_explainability(model, dataset)
            elif reg_name == 'data_governance':
                result = test_data_governance(model, dataset)

            compliance_results[reg_name] = result

        return compliance_results

# Define stress scenarios
stress_scenarios = {
    'recession': {'gdp_growth': -0.05, 'unemployment': 0.10},
    'inflation': {'inflation_rate': 0.08, 'interest_rate': 0.06},
    'market_crash': {'stock_market_change': -0.30, 'volatility': 0.50}
}

# Run custom tests
financial_tests = CustomFinancialTests()
stress_results = financial_tests.test_economic_stress(
    giskard_model, giskard_dataset, stress_scenarios
)

Performance Optimization

1. Efficient Scanning

# Optimize scan performance
def optimize_giskard_scan(model, dataset):
    """Optimize Giskard scan for large datasets."""

    # Use sampling for large datasets
    if len(dataset.df) > 10000:
        sample_size = min(5000, len(dataset.df))
        sampled_indices = np.random.choice(
            len(dataset.df), sample_size, replace=False
        )
        sampled_dataset = giskard.Dataset(
            df=dataset.df.iloc[sampled_indices],
            target=dataset.target.iloc[sampled_indices] if hasattr(dataset.target, 'iloc') else dataset.target[sampled_indices],
            name=f"Sampled_{dataset.name}"
        )
        dataset = sampled_dataset

    # Run focused scan on critical issues first
    critical_scan = giskard.scan(
        model, dataset,
        only=['performance', 'bias'],
        max_issues_per_detector=3
    )

    # If critical issues found, run full scan
    if any(issue.level == 'high' for issue in critical_scan.issues):
        full_scan = giskard.scan(model, dataset)
        return full_scan
    else:
        return critical_scan

# Usage
optimized_results = optimize_giskard_scan(giskard_model, giskard_dataset)

2. Parallel Testing

import concurrent.futures
from functools import partial

def run_parallel_tests(models_and_datasets):
    """Run Giskard tests in parallel for multiple models."""

    def run_single_test(model_dataset_pair):
        model, dataset, name = model_dataset_pair
        try:
            results = giskard.scan(model, dataset)
            return (name, results)
        except Exception as e:
            return (name, {'error': str(e)})

    # Run tests in parallel
    with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
        futures = [
            executor.submit(run_single_test, pair)
            for pair in models_and_datasets
        ]

        results = {}
        for future in concurrent.futures.as_completed(futures):
            name, result = future.result()
            results[name] = result

    return results

# Example usage
models_and_datasets = [
    (model1, dataset1, "Model_A"),
    (model2, dataset2, "Model_B"),
    (model3, dataset3, "Model_C")
]

parallel_results = run_parallel_tests(models_and_datasets)

Troubleshooting and Common Issues

Common Installation Issues

# Issue: Dependency conflicts
# Solution: Create clean environment
conda create -n giskard-env python=3.8
conda activate giskard-env
pip install giskard

# Issue: Missing dependencies for specific model types
# Solution: Install with extras
pip install "giskard[all]"  # All dependencies
pip install "giskard[nlp,vision]"  # Specific types

# Issue: CUDA/GPU issues
# Solution: Install CPU-only versions
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install tensorflow-cpu

Model Wrapping Issues

# Issue: Model prediction format problems
def fix_prediction_format(model_predictions):
    """Standardize model prediction format for Giskard."""
    if len(model_predictions.shape) == 2:
        # For multi-class classification, return class indices
        return np.argmax(model_predictions, axis=1)
    elif isinstance(model_predictions[0], str):
        # For string labels, convert to indices
        unique_labels = np.unique(model_predictions)
        label_to_idx = {label: idx for idx, label in enumerate(unique_labels)}
        return np.array([label_to_idx[label] for label in model_predictions])
    else:
        return model_predictions

# Wrap model with format fixing
def wrapped_predict(df):
    raw_predictions = original_model.predict(df)
    return fix_prediction_format(raw_predictions)

giskard_model = giskard.Model(
    model=wrapped_predict,
    model_type="classification",
    feature_names=feature_names
)

Performance Issues

# Issue: Slow scanning on large datasets
# Solution: Progressive scanning approach
def progressive_scan(model, dataset, max_size=5000):
    """Implement progressive scanning for large datasets."""

    if len(dataset.df) <= max_size:
        return giskard.scan(model, dataset)

    # Start with small sample
    sample_sizes = [1000, 2500, 5000]
    for size in sample_sizes:
        if size >= len(dataset.df):
            size = len(dataset.df)

        # Random sample
        sample_indices = np.random.choice(len(dataset.df), size, replace=False)
        sample_dataset = giskard.Dataset(
            df=dataset.df.iloc[sample_indices],
            target=dataset.target.iloc[sample_indices],
            name=f"Progressive_Sample_{size}"
        )

        # Quick scan
        results = giskard.scan(
            model, sample_dataset,
            max_issues_per_detector=2
        )

        # If no critical issues, continue with larger sample
        critical_issues = [i for i in results.issues if i.level == 'high']
        if not critical_issues and size < len(dataset.df):
            continue
        else:
            return results

    # Full scan if needed
    return giskard.scan(model, dataset)

# Usage
progressive_results = progressive_scan(giskard_model, giskard_dataset)

Advanced Use Cases

Multi-Model Testing

# Compare multiple models systematically
def compare_models(models_dict, dataset):
    """Compare multiple models using Giskard."""

    comparison_results = {}

    for model_name, model in models_dict.items():
        # Wrap model for Giskard
        giskard_model = giskard.Model(
            model=model,
            model_type="classification",
            feature_names=dataset.df.columns.tolist(),
            name=model_name
        )

        # Run comprehensive scan
        scan_results = giskard.scan(giskard_model, dataset)

        # Extract key metrics
        comparison_results[model_name] = {
            'total_issues': len(scan_results.issues),
            'critical_issues': len([i for i in scan_results.issues if i.level == 'high']),
            'bias_issues': len([i for i in scan_results.issues if 'bias' in i.issue_type]),
            'performance_issues': len([i for i in scan_results.issues if 'performance' in i.issue_type]),
            'robustness_score': calculate_robustness_score(scan_results),
            'overall_score': calculate_overall_score(scan_results)
        }

    return comparison_results

# Define models to compare
models_to_compare = {
    'Random_Forest': RandomForestClassifier(),
    'XGBoost': XGBClassifier(),
    'Neural_Network': MLPClassifier(),
    'Logistic_Regression': LogisticRegression()
}

# Train all models (simplified)
for name, model in models_to_compare.items():
    model.fit(X_train, y_train)

# Compare models
model_comparison = compare_models(models_to_compare, giskard_dataset)

# Display comparison results
comparison_df = pd.DataFrame(model_comparison).T
print(comparison_df)

Continuous Learning Integration

class ContinuousLearningMonitor:
    """Monitor model quality during continuous learning."""

    def __init__(self, initial_model, reference_dataset):
        self.current_model = initial_model
        self.reference_dataset = reference_dataset
        self.quality_history = []

    def evaluate_model_update(self, new_model, new_data_batch):
        """Evaluate quality impact of model update."""

        # Test current model on new data
        current_results = giskard.scan(
            self.current_model,
            new_data_batch
        )

        # Test new model on reference data
        new_model_results = giskard.scan(
            new_model,
            self.reference_dataset
        )

        # Test new model on new data
        new_on_new_results = giskard.scan(
            new_model,
            new_data_batch
        )

        # Compare quality metrics
        quality_comparison = {
            'timestamp': datetime.now(),
            'current_on_new': self._extract_metrics(current_results),
            'new_on_reference': self._extract_metrics(new_model_results),
            'new_on_new': self._extract_metrics(new_on_new_results),
            'recommendation': self._make_recommendation(
                current_results, new_model_results, new_on_new_results
            )
        }

        self.quality_history.append(quality_comparison)
        return quality_comparison

    def _extract_metrics(self, scan_results):
        return {
            'total_issues': len(scan_results.issues),
            'critical_issues': len([i for i in scan_results.issues if i.level == 'high']),
            'quality_score': self._calculate_quality_score(scan_results)
        }

    def _make_recommendation(self, current_results, new_ref_results, new_new_results):
        """Make recommendation about model update."""

        current_critical = len([i for i in current_results.issues if i.level == 'high'])
        new_ref_critical = len([i for i in new_ref_results.issues if i.level == 'high'])
        new_new_critical = len([i for i in new_new_results.issues if i.level == 'high'])

        if new_ref_critical > current_critical + 2:
            return "REJECT: New model significantly worse on reference data"
        elif new_new_critical > 5:
            return "REJECT: New model has too many issues on new data"
        elif new_ref_critical <= current_critical and new_new_critical <= 2:
            return "ACCEPT: New model shows improvement or maintains quality"
        else:
            return "REVIEW: Manual review recommended"

    def update_model(self, new_model):
        """Update current model after quality approval."""
        self.current_model = new_model

# Usage
monitor = ContinuousLearningMonitor(initial_model, reference_dataset)

# Simulate continuous learning cycle
for batch in new_data_batches:
    # Train updated model (simplified)
    updated_model = retrain_model(current_model, batch)

    # Evaluate quality impact
    evaluation = monitor.evaluate_model_update(updated_model, batch)

    print(f"Quality evaluation: {evaluation['recommendation']}")

    # Update model if recommended
    if evaluation['recommendation'].startswith('ACCEPT'):
        monitor.update_model(updated_model)

Conclusion

Giskard AI represents a significant advancement in ML model testing and quality assurance. Its comprehensive approach to vulnerability detection, combined with deep integration capabilities and domain-specific testing features, makes it an essential tool for any organization serious about AI quality and reliability.

Key Benefits:

  • Comprehensive Testing: Automated detection of diverse ML vulnerabilities
  • Easy Integration: Seamless integration with existing ML workflows
  • Production Ready: Continuous monitoring and alerting capabilities
  • Domain Expertise: Specialized testing for different application areas
  • Open Source: Transparent, extensible, and community-driven development

Getting Started:

  1. Install Giskard: pip install giskard
  2. Wrap Your Model: Create Giskard model and dataset objects
  3. Run Initial Scan: giskard.scan(model, dataset)
  4. Analyze Results: Review and prioritize detected issues
  5. Integrate: Add quality gates to your ML pipeline
  6. Monitor: Set up continuous production monitoring

Giskard AI empowers teams to build more reliable, fair, and robust AI systems through systematic quality assurance practices. Start your journey toward higher-quality AI systems today.

Ready to improve your AI model quality? Begin with our Giskard Quick Start Guide and transform your approach to ML testing.

← Back to The AI Tester's KitBrowse All Resources