← Back to Contents

Chapter 07: Cross-Device vs Cross-Silo Federated Learning

Compare and optimize for different federated learning deployment scenarios

Two Paradigms of Federated Learning

Federated learning manifests in two distinct paradigms with fundamentally different characteristics, challenges, and solutions. Cross-device FL involves millions of mobile devices or IoT sensors, while cross-silo FL connects a small number of organizations or data centers. Understanding these differences is crucial for designing effective FL systems.

Cross-Device Federated Learning

Characteristics

• Scale: Millions to billions of clients (10⁶-10¹⁰)
• Client Reliability: Highly unreliable, frequent dropouts
• Data per Client: Small datasets (100s-1000s of samples)
• Availability: Intermittent, unpredictable schedules
• Resources: Limited compute, memory, battery, bandwidth
• Network: Slow, variable, expensive (cellular)
• Privacy Sensitivity: Very high (personal data)

Example Applications

• Mobile keyboard next-word prediction (Gboard)
• Voice assistant personalization
• Health tracking apps
• Photo tagging on smartphones
• Mobile game recommendation

Technical Approach

class CrossDeviceFederatedLearning:
    """
    FL system optimized for cross-device scenario

    Key features:
    - Handles massive client population
    - Tolerates high dropout rates
    - Minimizes communication cost
    - Works with limited device resources
    """

    def __init__(self):
        self.total_clients = 10_000_000  # 10 million devices
        self.clients_per_round = 100      # Sample 100 per round
        self.local_epochs = 1             # Minimal local training
        self.batch_size = 10              # Small batches
        self.dropout_tolerance = 0.5      # Expect 50% dropout

        # Aggressive compression for mobile networks
        self.compressor = GradientCompressionPipeline(
            quantization_bits=8,
            sparsity=0.01,  # Keep only 1% of gradients
            use_error_feedback=True
        )

    def select_clients(self):
        """
        Sample clients for this round

        Criteria:
        - Device is idle (screen off, charging)
        - Connected to WiFi
        - Battery > 30%
        - Not in low-power mode
        """
        eligible_clients = self.filter_by_device_state()

        # Over-sample to account for dropouts
        num_to_select = int(self.clients_per_round / (1 - self.dropout_tolerance))

        return random.sample(eligible_clients, num_to_select)

    def client_update(self, client_id):
        """
        Minimal local training to conserve resources

        - Single epoch
        - Small batches
        - Aggressive compression
        """
        local_data = load_client_data(client_id)

        # Quick local training
        model_update = train_one_epoch(local_data, batch_size=self.batch_size)

        # Compress heavily for transmission
        compressed_update = self.compressor.compress(model_update)

        return compressed_update

    def handle_stragglers(self, client_updates, timeout=300):
        """
        Don't wait for slow clients

        Aggregate as soon as minimum number respond
        """
        min_clients = int(self.clients_per_round * 0.5)

        updates_received = []
        start_time = time.time()

        while len(updates_received) < self.clients_per_round:
            # Check for new updates
            new_updates = poll_client_responses()
            updates_received.extend(new_updates)

            # Time out if taking too long or have minimum
            if time.time() - start_time > timeout and len(updates_received) >= min_clients:
                break

        return updates_received

Cross-Silo Federated Learning

Characteristics

• Scale: Few to hundreds of clients (2-100)
• Client Reliability: Highly reliable, stable connections
• Data per Client: Large datasets (millions of samples)
• Availability: Predictable, scheduled participation
• Resources: Abundant compute (GPUs/TPUs), memory, bandwidth
• Network: Fast, stable, private (VPN/dedicated links)
• Privacy Sensitivity: Extremely high (competitive/regulatory)

Example Applications

• Multi-hospital disease prediction
• Inter-bank fraud detection
• Cross-company supply chain optimization
• Government agency data sharing
• Research institution collaboration

Technical Approach

class CrossSiloFederatedLearning:
    """
    FL system optimized for cross-silo scenario

    Key features:
    - Small number of powerful participants
    - Strong privacy guarantees (secure MPC)
    - Handles large datasets per silo
    - Sophisticated aggregation algorithms
    """

    def __init__(self):
        self.num_silos = 10               # 10 hospitals/banks/companies
        self.clients_per_round = 8        # Most participate each round
        self.local_epochs = 10            # Extensive local training
        self.batch_size = 256             # Large batches

        # Strong privacy mechanisms
        self.privacy_mechanism = SecureMultiPartyComputation()
        self.differential_privacy = DifferentialPrivacy(epsilon=0.5)

        # Sophisticated aggregation for non-IID data
        self.aggregator = SCAFFOLD()  # Variance reduction

    def coordinate_training_round(self):
        """
        Scheduled coordination among silos

        All silos participate in each round (high reliability)
        """
        # Send notification to all silos
        self.notify_silos_to_begin()

        # Wait for all silos (they're reliable)
        silo_updates = self.collect_all_silo_updates(timeout=3600)  # 1 hour timeout

        # Verify all expected silos participated
        if len(silo_updates) < self.num_silos * 0.8:  # At least 80%
            logging.warning("Some silos did not participate")

        return silo_updates

    def secure_aggregation(self, silo_updates):
        """
        Use secure MPC to aggregate without revealing individual updates

        Each silo's contribution remains encrypted during aggregation
        """
        # Each silo encrypts their update
        encrypted_updates = []
        for silo_id, update in silo_updates.items():
            encrypted = self.privacy_mechanism.encrypt(update, silo_id)
            encrypted_updates.append(encrypted)

        # Aggregate encrypted updates (homomorphic)
        encrypted_result = self.privacy_mechanism.aggregate(encrypted_updates)

        # Decrypt final result (requires threshold of silos)
        aggregated_model = self.privacy_mechanism.decrypt(encrypted_result)

        # Add differential privacy noise
        private_model = self.differential_privacy.add_noise(aggregated_model)

        return private_model

    def handle_non_iid_data(self):
        """
        Address statistical heterogeneity across silos

        Each hospital/bank has very different data distributions
        """
        # Use advanced algorithms
        # - FedProx: Proximal term regularization
        # - SCAFFOLD: Control variates
        # - FedBN: Separate batch normalization per silo

        return "Use sophisticated methods for non-IID data"

Detailed Comparison

Aspect	Cross-Device	Cross-Silo
Number of Clients	10⁶ - 10¹⁰	2 - 100
Participation Rate	<1% per round	50-100% per round
Dataset Size/Client	Small (KB-MB)	Large (GB-TB)
Client Reliability	Low (frequent dropouts)	High (stable)
Communication Cost	Critical bottleneck	Less critical
Computation/Client	Limited (mobile CPU)	Abundant (GPUs/TPUs)
Privacy Requirements	High (personal data)	Very high (competitive/legal)
Data Distribution	Highly non-IID	Extremely non-IID
Coordination	Asynchronous	Synchronous (scheduled)
Training Speed	Slow (many rounds)	Faster (fewer rounds)

Design Principles for Each Paradigm

Cross-Device Principles

1. Minimize Communication: Use aggressive compression (100x+)
2. Handle Dropouts: Don't wait for stragglers, over-sample clients
3. Conserve Resources: Run during idle, charging, WiFi conditions
4. Scale Massively: Server must handle millions of potential clients
5. Simple Algorithms: Complexity limited by device capabilities
6. Privacy by Design: Never send raw data, use local DP

Cross-Silo Principles

1. Strong Privacy: Use secure MPC, homomorphic encryption
2. Maximize Utility: Less compression, more local training
3. Sophisticated Aggregation: Handle extreme non-IID with advanced methods
4. Reliability: Expect all silos to participate, schedule coordination
5. Compliance: Meet regulatory requirements (GDPR, HIPAA)
6. Fairness: Ensure equitable benefit distribution among silos

Hybrid Approaches

Hierarchical Federated Learning

Combine cross-device and cross-silo characteristics:

class HierarchicalFederatedLearning:
    """
    Three-tier architecture combining both paradigms

    Tier 1: Mobile devices (cross-device)
    Tier 2: Edge servers (aggregation)
    Tier 3: Cloud data centers (cross-silo)
    """

    def __init__(self):
        self.edge_servers = 100        # Regional edge servers
        self.devices_per_edge = 10000  # Devices per edge server
        self.central_cloud = 1         # Global aggregation

    def training_round(self):
        """
        Multi-level aggregation

        1. Devices → Edge servers (cross-device style)
        2. Edge servers → Cloud (cross-silo style)
        """

        # Level 1: Cross-device FL at each edge
        edge_models = []
        for edge_server in self.edge_servers:
            # Sample local devices
            local_devices = edge_server.sample_devices(num=100)

            # Aggregate locally (cross-device methods)
            edge_model = edge_server.aggregate_devices(
                local_devices,
                compression='aggressive',
                dropout_tolerance=0.5
            )

            edge_models.append(edge_model)

        # Level 2: Cross-silo FL among edge servers
        global_model = self.central_cloud.aggregate_edges(
            edge_models,
            method='secure_mpc',
            privacy='strong_dp'
        )

        return global_model

Choosing the Right Approach

Implementation Example: Healthcare Cross-Silo

class MedicalFederatedLearning:
    """
    Cross-silo FL for multi-hospital disease prediction

    Hospitals: 20 participating institutions
    Data: Patient records (HIPAA-protected)
    Goal: Collaborative model without sharing patient data
    """

    def __init__(self, hospitals):
        self.hospitals = hospitals  # List of hospital clients
        self.num_hospitals = len(hospitals)

        # HIPAA-compliant privacy
        self.secure_mpc = SecureMultiPartyComputation()
        self.differential_privacy = DifferentialPrivacy(epsilon=0.3)  # Strong privacy

        # Handle non-IID medical data
        self.aggregator = FedProx(mu=0.01)

    def collaborative_training(self, num_rounds=100):
        """
        Train disease prediction model across hospitals
        """
        global_model = initialize_model()

        for round_num in range(num_rounds):
            print(f"Round {round_num + 1}/{num_rounds}")

            # All hospitals participate (scheduled)
            hospital_updates = {}

            for hospital in self.hospitals:
                # Each hospital trains locally on their patient data
                local_model = hospital.train_on_local_data(
                    global_model,
                    epochs=10,
                    batch_size=128
                )

                # Compute model update
                update = local_model - global_model

                # Clip and add local DP for patient privacy
                update = clip_and_add_noise(update, epsilon=0.1)

                hospital_updates[hospital.id] = update

            # Secure aggregation (encrypted)
            global_model = self.secure_mpc.aggregate_encrypted(
                hospital_updates,
                global_model
            )

            # Server-side DP for additional protection
            global_model = self.differential_privacy.add_noise(global_model)

            # Evaluate on validation data
            metrics = self.evaluate_model(global_model)
            print(f"Accuracy: {metrics['accuracy']:.3f}, AUC: {metrics['auc']:.3f}")

        return global_model

    def ensure_fairness(self, hospital_contributions, global_model):
        """
        Ensure all hospitals benefit from collaboration

        Measure: Each hospital's model should improve
        """
        for hospital in self.hospitals:
            local_only_accuracy = hospital.evaluate_local_model()
            federated_accuracy = hospital.evaluate_model(global_model)

            improvement = federated_accuracy - local_only_accuracy

            assert improvement >= 0, f"{hospital.name} did not benefit from federation"

            print(f"{hospital.name}: {improvement:.2%} improvement")