HTTP/2 Denial of Service Advisory

Overview

In August, 2019 Netflix discovered several resource exhaustion vectors that could be used to launch denial of service (DoS) attacks against servers supporting HTTP/2 communication. Netflix worked with Google and CERT/CC to coordinate disclosure to the Internet community. For more information read the 13Aug2019 denial of service security advisory for HTTP/2 advisory. Starting with Avi Vantage release 18.2.6, four new parameters have been incorporated to cover these security issues.

Common Vulnerabilities and Exposures

The following 8 CVEs are associated with this advisory, as documented in the advisory mentioned above.

  • CVE-2019-9511: HTTP/2 Data Dribble
  • CVE-2019-9512: HTTP/2 Ping Flood
  • CVE-2019-9513: HTTP/2 Resource Loop
  • CVE-2019-9514: HTTP/2 Reset Flood
  • CVE-2019-9515: HTTP/2 Settings Flood
  • CVE-2019-9516: HTTP/2 0-Length Headers Leak
  • CVE-2019-9517: HTTP/2 Internal Data Buffering
  • CVE-2019-9518: HTTP/2 Empty Frame Flooding

Addressing the CVEs and Exposures

The above CVEs and other issues are addressed via new HTTP-application-profile parameters and other code. At the time of this writing, the four parameters are accessible via the Avi REST API and Avi CLI. Avi UI access is not supported.

  • max_http2_control_frames_per_connection
    • Maximum number of control frames that a client can send over an HTTP/2 connection
    • Used to detect ping flood (CVE-2019-9512), reset flood (CVE-2019-9514),setting flood (CVE-2019-9515) and resource loop issue (CVE-2019-9513)
    • Ranges from 0 - 10,000; default = 1000
    • 0 is interpreted as unlimited control frames on a client-side HTTP/2 connection
  • max_http2_queued_frames_to_client_per_connection
    • Maximum number of frames that can be queued waiting to be sent over a client-side HTTP/2 connection at any given time
    • Used to detect data dribble (CVE-2019-9511 ) and internal data buffering (CVE-2019-9517)
    • Ranges from 0 - 10,000; default = 1000
    • 0 is interpreted as unlimited queued frames
    • Also incorporated is a fix from NGINX, nginx/nginx@a987f81
  • max_http2_empty_data_frames_per_connection
    • Maximum number of empty data frames over a client-side HTTP/2 connection
    • Used to detect empty data frame flooding (CVE-2019-9518)

      Note: For an empty header frame, our current implementation closes the connection and and reports an error like: HTTP2 Frame Size Error: Client sent HEADERS frame with empty header block.

    • Ranges from 0 - 10,000; default = 1000
    • 0 is interpreted as unlimited empty frames
  • max_http2_concurrent_streams_per_connection
    • Maximum number of concurrent streams over a client-side HTTP/2 connection
    • Used to control the concurrent streams one HTTP/2 connection can handle at any given time
    • Ranges from 1 - 256; default = 128

      Note: This parameter is not related to any of the CVEs issues mentioned above.

The 0-length headers leak (CVE-2019-9516) is detected by current code. Upon detection, the stream will be closed and Avi will report an error like: HTTP2 Protocol Error: Client sent zero length header name.

Avi CLI Configuration Example

[admin:th-controller-3]: > show applicationprofile  applicationprofile-22
+----------------------------------------------------+---------------------------------------------------------+
| Field                                              | Value                                                   |
+----------------------------------------------------+---------------------------------------------------------+
| uuid                                               | applicationprofile-8cd83b63-9ef8-4ce3-b498-2d173a09ed7c |
| name                                               | applicationprofile-22                                   |
| type                                               | APPLICATION_PROFILE_TYPE_HTTP                           |
| http_profile                                       |                                                         |
|   connection_multiplexing_enabled                  | True                                                    |
|   xff_enabled                                      | False                                                   |
|   xff_alternate_name                               | X-Forwarded-For                                         |
|   ssl_everywhere_enabled                           | False                                                   |
|   hsts_enabled                                     | False                                                   |
|   hsts_max_age                                     | 365                                                     |
|   secure_cookie_enabled                            | False                                                   |
|   httponly_enabled                                 | False                                                   |
|   http_to_https                                    | False                                                   |
|   server_side_redirect_to_https                    | False                                                   |
|   x_forwarded_proto_enabled                        | False                                                   |
|   spdy_enabled                                     | False                                                   |
|   spdy_fwd_proxy_mode                              | False                                                   |
|   post_accept_timeout                              | 30000 milliseconds                                      |
|   client_header_timeout                            | 10000 milliseconds                                      |
|   client_body_timeout                              | 30000 milliseconds                                      |
|   keepalive_timeout                                | 30000 milliseconds                                      |
|   client_max_header_size                           | 12 kb                                                   |
|   client_max_request_size                          | 48 kb                                                   |
|   client_max_body_size                             | 0 kb                                                    |
|   max_rps_unknown_uri                              | 0                                                       |
|   max_rps_cip                                      | 0                                                       |
|   max_rps_uri                                      | 0                                                       |
|   max_rps_cip_uri                                  | 0                                                       |
|   ssl_client_certificate_mode                      | SSL_CLIENT_CERTIFICATE_NONE                             |
|   websockets_enabled                               | True                                                    |
|   max_rps_unknown_cip                              | 0                                                       |
|   max_bad_rps_cip                                  | 0                                                       |
|   max_bad_rps_uri                                  | 0                                                       |
|   max_bad_rps_cip_uri                              | 0                                                       |
|   keepalive_header                                 | False                                                   |
|   use_app_keepalive_timeout                        | False                                                   |
|   allow_dots_in_header_name                        | False                                                   |
|   disable_keepalive_posts_msie6                    | True                                                    |
|   enable_request_body_buffering                    | False                                                   |
|   enable_fire_and_forget                           | False                                                   |
|   max_response_headers_size                        | 48 kb                                                   |
|   http2_enabled                                    | True                                                    |
|   respond_with_100_continue                        | True                                                    |
|   hsts_subdomains_enabled                          | True                                                    |
|   enable_request_body_metrics                      | False                                                   |
|   fwd_close_hdr_for_bound_connections              | True                                                    |
|   max_keepalive_requests                           | 100                                                     |
|   disable_sni_hostname_check                       | False                                                   |
|   max_http2_control_frames_per_connection          | 1000                                                    |
|   max_http2_queued_frames_to_client_per_connection | 1000                                                    |
|   max_http2_empty_data_frames_per_connection       | 1000                                                    |
|   max_http2_concurrent_streams_per_connection      | 128                                                     |
|   reset_conn_http_on_ssl_port                      | False                                                   |
| preserve_client_ip                                 | False                                                   |
| preserve_client_port                               | False                                                   |
| tenant_ref                                         | admin                                                   |
+----------------------------------------------------+---------------------------------------------------------+
[admin:th-controller-3]: >
[admin:th-controller-3]: applicationprofile> configure applicationprofile applicationprofile-22
[admin:th-controller-3]: applicationprofile> http_profile
[admin:th-controller-3]: applicationprofile:http_profile> max_http2_control_frames_per_connection 2000
Overwriting the previously entered value for max_http2_control_frames_per_connection
[admin:th-controller-3]: applicationprofile:http_profile> max_http2_queued_frames_to_client_per_connection 2000
Overwriting the previously entered value for max_http2_queued_frames_to_client_per_connection
[admin:th-controller-3]: applicationprofile:http_profile> max_http2_empty_data_frames_per_connection 2000
Overwriting the previously entered value for max_http2_empty_data_frames_per_connection
[admin:th-controller-3]: applicationprofile:http_profile> max_http2_concurrent_streams_per_connection 256
Overwriting the previously entered value for max_http2_concurrent_streams_per_connection
[admin:th-controller-3]: applicationprofile:http_profile> save
[admin:th-controller-3]: applicationprofile> save
[admin:th-controller-3]: > show applicationprofile applicationprofile-22
+----------------------------------------------------+---------------------------------------------------------+
| Field                                              | Value                                                   |
+----------------------------------------------------+---------------------------------------------------------+
| uuid                                               | applicationprofile-8cd83b63-9ef8-4ce3-b498-2d173a09ed7c |
| name                                               | applicationprofile-22                                   |
| type                                               | APPLICATION_PROFILE_TYPE_HTTP                           |
| http_profile                                       |                                                         |
|   connection_multiplexing_enabled                  | True                                                    |
|   xff_enabled                                      | False                                                   |
|   xff_alternate_name                               | X-Forwarded-For                                         |
|   ssl_everywhere_enabled                           | False                                                   |
|   hsts_enabled                                     | False                                                   |
|   hsts_max_age                                     | 365                                                     |
|   secure_cookie_enabled                            | False                                                   |
|   httponly_enabled                                 | False                                                   |
|   http_to_https                                    | False                                                   |
|   server_side_redirect_to_https                    | False                                                   |
|   x_forwarded_proto_enabled                        | False                                                   |
|   spdy_enabled                                     | False                                                   |
|   spdy_fwd_proxy_mode                              | False                                                   |
|   post_accept_timeout                              | 30000 milliseconds                                      |
|   client_header_timeout                            | 10000 milliseconds                                      |
|   client_body_timeout                              | 30000 milliseconds                                      |
|   keepalive_timeout                                | 30000 milliseconds                                      |
|   client_max_header_size                           | 12 kb                                                   |
|   client_max_request_size                          | 48 kb                                                   |
|   client_max_body_size                             | 0 kb                                                    |
|   max_rps_unknown_uri                              | 0                                                       |
|   max_rps_cip                                      | 0                                                       |
|   max_rps_uri                                      | 0                                                       |
|   max_rps_cip_uri                                  | 0                                                       |
|   ssl_client_certificate_mode                      | SSL_CLIENT_CERTIFICATE_NONE                             |
|   websockets_enabled                               | True                                                    |
|   max_rps_unknown_cip                              | 0                                                       |
|   max_bad_rps_cip                                  | 0                                                       |
|   max_bad_rps_uri                                  | 0                                                       |
|   max_bad_rps_cip_uri                              | 0                                                       |
|   keepalive_header                                 | False                                                   |
|   use_app_keepalive_timeout                        | False                                                   |
|   allow_dots_in_header_name                        | False                                                   |
|   disable_keepalive_posts_msie6                    | True                                                    |
|   enable_request_body_buffering                    | False                                                   |
|   enable_fire_and_forget                           | False                                                   |
|   max_response_headers_size                        | 48 kb                                                   |
|   http2_enabled                                    | True                                                    |
|   respond_with_100_continue                        | True                                                    |
|   hsts_subdomains_enabled                          | True                                                    |
|   enable_request_body_metrics                      | False                                                   |
|   fwd_close_hdr_for_bound_connections              | True                                                    |
|   max_keepalive_requests                           | 100                                                     |
|   disable_sni_hostname_check                       | False                                                   |
|   max_http2_control_frames_per_connection          | 2000                                                    |
|   max_http2_queued_frames_to_client_per_connection | 2000                                                    |
|   max_http2_empty_data_frames_per_connection       | 2000                                                    |
|   max_http2_concurrent_streams_per_connection      | 256                                                     |
|   reset_conn_http_on_ssl_port                      | False                                                   |
| preserve_client_ip                                 | False                                                   |
| preserve_client_port                               | False                                                   |
| tenant_ref                                         | admin                                                   |
+----------------------------------------------------+---------------------------------------------------------+
[admin:th-controller-3]: >

Changes for Avi Vantage 20.1.1 Release

Starting with Avi Vantage release 20.1.1, the HTTP/2 configuration fields have been moved to a sub-field under http_profile called http2_profile. The following three new fields have been added to the http2_profile:

  • max_http2_requests_per_connection – This value controls the maximum number of that can be sent over a client-side HTTP/2 connection. If the value is set to 0, this means an unlimited number of requests can be sent over an HTTP/2 client-side connection.
  • max_http2_header_field – This field controls the maximum size (in BYTES) of the compressed request header field. The limit applies equally to both name and value of the request header. The range varies from 1-8192 BYTES. The default value is 4096 BYTES.
  • http2_initial_window_size – This field controls the window size of the initial flow control in HTTP/2 streams. The value for this field ranges from 64-32768 KB. The default value for this field is 64KB.