You manage two VPCs: VPC1 and VPC2, each with resources spread across two regions.

The problem description indicates that VMs in VPC2 receive traffic but their replies don't reach VPC1, especially when a VPN gateway fails. This strongly suggests an asymmetric routing issue, where VPC2's routing table might not be aware of all necessary routes to send return traffic to VPC1, particularly in a multi-region setup with failover. By default, VPC networks are in regional dynamic routing mode, meaning they only learn routes from Cloud Routers in the same region. To ensure that routes learned from one region (where the active VPN tunnel might be) are available globally across the VPC, you need to enable global dynamic routing mode in the VPC that is experiencing the return traffic issue (VPC2 in this case). This allows VPC2 to learn and apply routes from Cloud Routers in all regions, ensuring that even if a VPN tunnel fails in one region, the routes learned from the active tunnel in another region are still available for return traffic.

Exact Extract:

"A VPC network's dynamic routing mode controls whether routes learned by Cloud Routers in one region are available to VMs in other regions. By default, VPC networks are in regional dynamic routing mode, which means Cloud Routers in a region only advertise routes to and learn routes from other Cloud Routers in the same region. This can lead to asymmetric routing issues in multi-region deployments."

"To ensure that routes learned from Cloud Routers are propagated to all regions within a VPC network, you must set the dynamic routing mode to global."Reference: Google Cloud VPC Documentation - Dynamic routing mode