Which functionality does an NGFW use to determine whether new session setups are legitimate or...

To preventSYN flood attacks, the NGFW usesSYN cookiesto validate legitimate session establishment.

“SYN cookies allow the firewall to verify the legitimacy of new session requests without allocating resources until the handshake is completed. This prevents SYN flood attacks from exhausting system resources.”

(Source: Flood Protection Best Practices)

SYN cookies mitigate resource exhaustion by ensuring only legitimate connections are established.