{"id":269159,"date":"2023-10-05T13:07:01","date_gmt":"2023-10-05T18:07:01","guid":{"rendered":"https:\/\/www.webscale.com\/?p=269159"},"modified":"2023-12-29T15:30:52","modified_gmt":"2023-12-29T20:30:52","slug":"using-the-kubernetes-horizontal-pod-autoscaler-with-prometheus-custom-metrics","status":"publish","type":"post","link":"https:\/\/www.webscale.com\/blog\/using-the-kubernetes-horizontal-pod-autoscaler-with-prometheus-custom-metrics\/","title":{"rendered":"Using the Kubernetes Horizontal Pod Autoscaler with Prometheus\u2019 Custom Metrics"},"content":{"rendered":"<p><i><span style=\"font-weight: 400;\">With Kubernetes as the backbone of the CloudFlow Platform, we manage a large, complex network of globally distributed clusters with the goal of optimizing where and when our customers\u2019 workloads are running to suit the needs of their application, while maintaining operational efficiency of the platform.<\/span><\/i><\/p>\n<p><span style=\"font-weight: 400;\">In order to maximize efficiency, we need a mechanism to dynamically expand and contract resources based on the demands of our customers\u2019 applications at any given time. We rely on the Kubernetes Horizontal Pod Autoscaler (HPA) to scale services in a reliable, predictable, and controllable manner.<\/span><\/p>\n<p><b>How the Horizontal Pod Autoscaler Works<\/b><\/p>\n<p><span style=\"font-weight: 400;\">The HPA controller runs regularly to check if any adjustments to the system are required. <\/span><a href=\"https:\/\/kubernetes.io\/docs\/tasks\/run-application\/horizontal-pod-autoscale\/\" target=\"_blank\" rel=\"noopener\"><span style=\"font-weight: 400;\">During each run<\/span><\/a><span style=\"font-weight: 400;\">, the controller manager queries the resource utilization against the metrics specified in each HorizontalPodAutoscaler definition. The controller manager obtains the metrics from either the resource metrics API (for per-pod resource metrics), or the custom metrics API (for all other metrics).<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-269164\" src=\"https:\/\/www.webscale.com\/wp-content\/uploads\/2023\/10\/kubernetes-horizontal-pod-autoscaler-how-it-works.png\" alt=\"\" width=\"647\" height=\"518\" srcset=\"https:\/\/www.webscale.com\/wp-content\/uploads\/2023\/10\/kubernetes-horizontal-pod-autoscaler-how-it-works.png 647w, https:\/\/www.webscale.com\/wp-content\/uploads\/2023\/10\/kubernetes-horizontal-pod-autoscaler-how-it-works-480x384.png 480w\" sizes=\"(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 647px, 100vw\" \/><\/p>\n<p><span style=\"font-weight: 400;\">The HPA in its basic form is pretty powerful, but where the HPA starts to take on superhero status is when we start to introduce custom metrics to drive autoscaling intelligence.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">To start, there are three different variables that need to be set before using the HPA. You set these within the HPA configuration itself:<\/span><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\">maxReplicas<\/span><\/li>\n<li><span style=\"font-weight: 400;\">minReplicas<\/span><\/li>\n<li><span style=\"font-weight: 400;\">some type of resource, or something to scale on<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">Resource type can be memory or CPU, which generally isn\u2019t ideal for us to scale on because it can be extremely variable. Rather, we define and pull in custom metrics from Prometheus to get more granular in how we autoscale.<\/span><\/p>\n<p><b>Incorporating Custom Metrics from Prometheus<\/b><\/p>\n<p><span style=\"font-weight: 400;\">We use the <\/span><a href=\"https:\/\/github.com\/DirectXMan12\/k8s-prometheus-adapter\" target=\"_blank\" rel=\"noopener\"><span style=\"font-weight: 400;\">Prometheus Adapter for Kubernetes Metrics APIs<\/span><\/a><span style=\"font-weight: 400;\"> to access the custom metrics on which to autoscale.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Using <\/span><a href=\"https:\/\/github.com\/kubernetes-sigs\/prometheus-adapter\/blob\/master\/deploy\/manifests\/custom-metrics-config-map.yaml\" target=\"_blank\" rel=\"noopener\"><span style=\"font-weight: 400;\">custom-metrics-config-map.yaml<\/span><\/a><span style=\"font-weight: 400;\"> as a starting point, we can see that the default settings already include CPU and memory.<\/span><\/p>\n<pre style=\"color: #f8f8f2; background-color: #272822; -moz-tab-size: 4; -o-tab-size: 4; tab-size: 4;\"><code class=\"language-yaml\" data-lang=\"yaml\"><span style=\"color: #66d9ef;\">resourceRules<\/span>:\r\n  <span style=\"color: #66d9ef;\">cpu<\/span>:\r\n    <span style=\"color: #66d9ef;\">containerQuery<\/span>: sum(rate(container_cpu_usage_seconds_total{&lt;&lt;.LabelMatchers&gt;&gt;}[1m])) by (&lt;&lt;.GroupBy&gt;&gt;)\r\n    <span style=\"color: #66d9ef;\">nodeQuery<\/span>: sum(rate(container_cpu_usage_seconds_total{&lt;&lt;.LabelMatchers&gt;&gt;, id=<span style=\"color: #e6db74;\">'\/'<\/span>}[1m])) by (&lt;&lt;.GroupBy&gt;&gt;)\r\n    <span style=\"color: #66d9ef;\">resources<\/span>:\r\n      <span style=\"color: #66d9ef;\">overrides<\/span>:\r\n        <span style=\"color: #66d9ef;\">node<\/span>:\r\n          <span style=\"color: #66d9ef;\">resource<\/span>: node\r\n        <span style=\"color: #66d9ef;\">namespace<\/span>:\r\n          <span style=\"color: #66d9ef;\">resource<\/span>: namespace\r\n        <span style=\"color: #66d9ef;\">pod_name<\/span>:\r\n          <span style=\"color: #66d9ef;\">resource<\/span>: pod\r\n    <span style=\"color: #66d9ef;\">containerLabel<\/span>: container_name\r\n  <span style=\"color: #66d9ef;\">memory<\/span>:\r\n    <span style=\"color: #66d9ef;\">containerQuery<\/span>: sum(container_memory_working_set_bytes{&lt;&lt;.LabelMatchers&gt;&gt;}) by (&lt;&lt;.GroupBy&gt;&gt;)\r\n    <span style=\"color: #66d9ef;\">nodeQuery<\/span>: sum(container_memory_working_set_bytes{&lt;&lt;.LabelMatchers&gt;&gt;,id=<span style=\"color: #e6db74;\">'\/'<\/span>}) by (&lt;&lt;.GroupBy&gt;&gt;)\r\n    <span style=\"color: #66d9ef;\">resources<\/span>:\r\n      <span style=\"color: #66d9ef;\">overrides<\/span>:\r\n        <span style=\"color: #66d9ef;\">node<\/span>:\r\n          <span style=\"color: #66d9ef;\">resource<\/span>: node\r\n        <span style=\"color: #66d9ef;\">namespace<\/span>:\r\n          <span style=\"color: #66d9ef;\">resource<\/span>: namespace\r\n        <span style=\"color: #66d9ef;\">pod_name<\/span>:\r\n          <span style=\"color: #66d9ef;\">resource<\/span>: pod\r\n    <span style=\"color: #66d9ef;\">containerLabel<\/span>: container_name\r\n<\/code><\/pre>\n<p><span style=\"font-weight: 400;\">In this example, we\u2019ll add some extra parameters to the config map. These are just <\/span><a href=\"https:\/\/www.webscale.com\/blog\/prometheus-querying\/\"><span style=\"font-weight: 400;\">PromQL queries<\/span><\/a><span style=\"font-weight: 400;\"> with some substitutions that the Prometheus Adapter will fill in appropriately:<\/span><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\">Network received packets total &#8211; provided by the <\/span><a href=\"https:\/\/github.com\/prometheus\/node_exporter\" target=\"_blank\" rel=\"noopener\"><span style=\"font-weight: 400;\">Node Exporter<\/span><\/a><\/li>\n<li><span style=\"font-weight: 400;\">Request count &#8211; a custom metric that our pods expose for scraping to Prometheus. Not all our Pods have this metric, so packets received makes a decent proxy.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">Here\u2019s what it looks like:<\/span><\/p>\n<pre style=\"color: #f8f8f2; background-color: #272822; -moz-tab-size: 4; -o-tab-size: 4; tab-size: 4;\"><code class=\"language-yaml\" data-lang=\"yaml\"><span style=\"color: #66d9ef;\">rules<\/span>:\r\n- <span style=\"color: #66d9ef;\">seriesQuery<\/span>: <span style=\"color: #e6db74;\">'container_network_receive_packets_total{namespace!=\"\",pod_name!=\"\"}'<\/span>\r\n  <span style=\"color: #66d9ef;\">resources<\/span>:\r\n    <span style=\"color: #66d9ef;\">overrides<\/span>:\r\n      <span style=\"color: #66d9ef;\">namespace<\/span>: {<span style=\"color: #66d9ef;\">resource<\/span>: <span style=\"color: #e6db74;\">\"namespace\"<\/span>}\r\n      <span style=\"color: #66d9ef;\">pod_name<\/span>: {<span style=\"color: #66d9ef;\">resource<\/span>: <span style=\"color: #e6db74;\">\"pod\"<\/span>}\r\n  <span style=\"color: #66d9ef;\">name<\/span>:\r\n    <span style=\"color: #66d9ef;\">matches<\/span>: <span style=\"color: #e6db74;\">\"^(.*)\"<\/span>\r\n    <span style=\"color: #66d9ef;\">as<\/span>: <span style=\"color: #e6db74;\">\"pps1m\"<\/span>\r\n  <span style=\"color: #66d9ef;\">metricsQuery<\/span>: <span style=\"color: #e6db74;\">'sum(rate(&lt;&lt;.Series&gt;&gt;{&lt;&lt;.LabelMatchers&gt;&gt;}[1m])) by (&lt;&lt;.GroupBy&gt;&gt;)'<\/span>\r\n- <span style=\"color: #66d9ef;\">seriesQuery<\/span>: <span style=\"color: #e6db74;\">'container_network_receive_packets_total{namespace!=\"\",pod_name!=\"\"}'<\/span>\r\n  <span style=\"color: #66d9ef;\">resources<\/span>:\r\n    <span style=\"color: #66d9ef;\">overrides<\/span>:\r\n      <span style=\"color: #66d9ef;\">namespace<\/span>: {<span style=\"color: #66d9ef;\">resource<\/span>: <span style=\"color: #e6db74;\">\"namespace\"<\/span>}\r\n      <span style=\"color: #66d9ef;\">pod_name<\/span>: {<span style=\"color: #66d9ef;\">resource<\/span>: <span style=\"color: #e6db74;\">\"pod\"<\/span>}\r\n  <span style=\"color: #66d9ef;\">name<\/span>:\r\n    <span style=\"color: #66d9ef;\">matches<\/span>: <span style=\"color: #e6db74;\">\"^(.*)\"<\/span>\r\n    <span style=\"color: #66d9ef;\">as<\/span>: <span style=\"color: #e6db74;\">\"pps20m\"<\/span>\r\n  <span style=\"color: #66d9ef;\">metricsQuery<\/span>: <span style=\"color: #e6db74;\">'sum(rate(&lt;&lt;.Series&gt;&gt;{&lt;&lt;.LabelMatchers&gt;&gt;}[20m])) by (&lt;&lt;.GroupBy&gt;&gt;)'<\/span>\r\n- <span style=\"color: #66d9ef;\">seriesQuery<\/span>: <span style=\"color: #e6db74;\">'section_http_request_count_total{namespace!=\"\",pod_name!=\"\"}'<\/span>\r\n  <span style=\"color: #66d9ef;\">resources<\/span>:\r\n    <span style=\"color: #66d9ef;\">overrides<\/span>:\r\n      <span style=\"color: #66d9ef;\">namespace<\/span>: {<span style=\"color: #66d9ef;\">resource<\/span>: <span style=\"color: #e6db74;\">\"namespace\"<\/span>}\r\n      <span style=\"color: #66d9ef;\">pod_name<\/span>: {<span style=\"color: #66d9ef;\">resource<\/span>: <span style=\"color: #e6db74;\">\"pod\"<\/span>}\r\n  <span style=\"color: #66d9ef;\">name<\/span>:\r\n    <span style=\"color: #66d9ef;\">matches<\/span>: <span style=\"color: #e6db74;\">\"^(.*)\"<\/span>\r\n    <span style=\"color: #66d9ef;\">as<\/span>: <span style=\"color: #e6db74;\">\"rps10m\"<\/span>\r\n  <span style=\"color: #66d9ef;\">metricsQuery<\/span>: <span style=\"color: #e6db74;\">'sum(rate(&lt;&lt;.Series&gt;&gt;{&lt;&lt;.LabelMatchers&gt;&gt;}[10m])) by (&lt;&lt;.GroupBy&gt;&gt;)'<\/span>\r\n- <span style=\"color: #66d9ef;\">seriesQuery<\/span>: <span style=\"color: #e6db74;\">'section_http_request_count_total{namespace!=\"\",pod_name!=\"\"}'<\/span>\r\n  <span style=\"color: #66d9ef;\">resources<\/span>:\r\n    <span style=\"color: #66d9ef;\">overrides<\/span>:\r\n      <span style=\"color: #66d9ef;\">namespace<\/span>: {<span style=\"color: #66d9ef;\">resource<\/span>: <span style=\"color: #e6db74;\">\"namespace\"<\/span>}\r\n      <span style=\"color: #66d9ef;\">pod_name<\/span>: {<span style=\"color: #66d9ef;\">resource<\/span>: <span style=\"color: #e6db74;\">\"pod\"<\/span>}\r\n  <span style=\"color: #66d9ef;\">name<\/span>:\r\n    <span style=\"color: #66d9ef;\">matches<\/span>: <span style=\"color: #e6db74;\">\"^(.*)\"<\/span>\r\n    <span style=\"color: #66d9ef;\">as<\/span>: <span style=\"color: #e6db74;\">\"rps1m\"<\/span>\r\n  <span style=\"color: #66d9ef;\">metricsQuery<\/span>: <span style=\"color: #e6db74;\">'sum(rate(&lt;&lt;.Series&gt;&gt;{&lt;&lt;.LabelMatchers&gt;&gt;}[1m])) by (&lt;&lt;.GroupBy&gt;&gt;)'<\/span>\r\n<\/code><\/pre>\n<p><span style=\"font-weight: 400;\">The Prometheus adapter has a custom language it uses where it will substitute in namespace or pod, which allows us to query the custom metrics API so the HPA can use it. <\/span><a href=\"https:\/\/github.com\/DirectXMan12\/k8s-prometheus-adapter\/blob\/master\/docs\/config-walkthrough.md\" target=\"_blank\" rel=\"noopener\"><span style=\"font-weight: 400;\">Read documentation<\/span><\/a><span style=\"font-weight: 400;\">.<\/span><\/p>\n<div class=\"highlight\">\n<pre style=\"color: #f8f8f2; background-color: #272822; -moz-tab-size: 4; -o-tab-size: 4; tab-size: 4;\"><code class=\"language-yaml\" data-lang=\"yaml\"><span style=\"color: #66d9ef;\">metricsQuery<\/span>: <span style=\"color: #e6db74;\">'sum(rate(&lt;&lt;.Series&gt;&gt;{&lt;&lt;.LabelMatchers&gt;&gt;}[1m])) by (&lt;&lt;.GroupBy&gt;&gt;)'<\/span>\r\n<\/code><\/pre>\n<\/div>\n<p><span style=\"font-weight: 400;\">Note, we can also rename the metric in the API with the following (seen in context above):<\/span><\/p>\n<div class=\"highlight\">\n<pre style=\"color: #f8f8f2; background-color: #272822; -moz-tab-size: 4; -o-tab-size: 4; tab-size: 4;\"><code class=\"language-yaml\" data-lang=\"yaml\"><span style=\"color: #66d9ef;\">name<\/span>:\r\n  <span style=\"color: #66d9ef;\">matches<\/span>: <span style=\"color: #e6db74;\">\"^(.*)\"<\/span>\r\n  <span style=\"color: #66d9ef;\">as<\/span>: <span style=\"color: #e6db74;\">\"rps1m\"<\/span>\r\n<\/code><\/pre>\n<\/div>\n<p><span style=\"font-weight: 400;\">In this example, we\u2019re pulling in Requests Per Second (RPS) for each of our <\/span><a href=\"https:\/\/www.section.io\/modules\/\" target=\"_blank\" rel=\"noopener\"><span style=\"font-weight: 400;\">edge modules<\/span><\/a><span style=\"font-weight: 400;\"> (e.g. WAF, caching, image optimization, etc.) so we can efficiently autoscale to accommodate the volume of requests at any given time.<\/span><\/p>\n<p><b>Observability with HPA and Custom Metrics<\/b><\/p>\n<p><span style=\"font-weight: 400;\">While the HPA is tremendously powerful, it does require significant, ongoing tuning to realize its benefits. To this end, we use Grafana dashboards for a convenient way to view HPA activity, specific to a particular pod or environment.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">In the example above, we can see an environment scaling up to ten replicas (from two) over a 24-hour window.<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-269163\" src=\"https:\/\/www.webscale.com\/wp-content\/uploads\/2023\/10\/hpa-grafana-dashboard-1024x531.png\" alt=\"\" width=\"1024\" height=\"531\" srcset=\"https:\/\/www.webscale.com\/wp-content\/uploads\/2023\/10\/hpa-grafana-dashboard-980x508.png 980w, https:\/\/www.webscale.com\/wp-content\/uploads\/2023\/10\/hpa-grafana-dashboard-480x249.png 480w\" sizes=\"(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) and (max-width: 980px) 980px, (min-width: 981px) 1024px, 100vw\" \/><\/p>\n<p><span style=\"font-weight: 400;\">It\u2019s important to note that each metric has its own scaling decision, which Kubernetes will collate and choose the highest value. For example, we might create a packets-per-second metric where we scale on a 20-minute window and on a 1-minute window. The 1-minute window can be useful to detect a big spike in traffic, whereas a 20-minute window can help indicate a longer-term traffic trend.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Whenever making custom settings, it\u2019s helpful to consider whether you\u2019re looking to adapt for spiky traffic or for a day\/night cycle traffic pattern.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Of course, your work is likely not done once you\u2019ve made a first pass at setting HPA values. Using a dashboard like the one above can help you adjust settings to suit your objectives. Do you want to run as few pods as possible to save money on nodes? Do you want to scale quickly to a maximum value? Appropriate settings will vary with the type of workload, and with ongoing software development, they may need adjusting over time.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>With Kubernetes as the backbone of the CloudFlow Platform, we manage a large, complex network of globally distributed clusters with the goal of optimizing where and when our customers\u2019 workloads are running to suit the needs of their application, while maintaining operational efficiency of the platform. In order to maximize efficiency, we need a mechanism [&hellip;]<\/p>\n","protected":false},"author":36,"featured_media":269163,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","rank_math_lock_modified_date":false,"_aioseo_description":"","_aioseo_keywords":"","_aioseo_og_article_section":"","_aioseo_og_article_tags":"","_aioseo_og_description":"","_aioseo_og_title":"","_aioseo_title":"","_aioseo_twitter_description":"","_aioseo_twitter_title":"","_author_photo":"field_6513304084a08","_doc_url":"","_dp_original":"269151","_et_autogenerated_title":"","_et_body_layout_enabled":"","_et_body_layout_id":"","_et_builder_dynamic_assets_loading_attr_threshold":"2","_et_builder_module_features_cache":null,"_et_builder_version":"","_et_default":"","_et_enabled":"","_et_footer_layout_enabled":"","_et_footer_layout_id":"","_et_header_layout_enabled":"","_et_header_layout_id":"","_et_pb_ab_current_shortcode":"","_et_pb_ab_subjects":"","_et_pb_built_for_post_type":"","_et_pb_custom_css":"","_et_pb_enable_shortcode_tracking":"","_et_pb_excluded_global_options":"","_et_pb_first_image":"","_et_pb_gutter_width":"","_et_pb_module_type":"","_et_pb_page_layout":"et_right_sidebar","_et_pb_page_z_index":"","_et_pb_post_hide_nav":"default","_et_pb_row_layout":"","_et_pb_show_page_creation":"","_et_pb_show_title":"on","_et_pb_side_nav":"off","_et_pb_static_css_file":"","_et_pb_truncate_post":"","_et_pb_truncate_post_date":"","_et_post_bg_color":"#ffffff","_et_post_bg_layout":"light","_et_template":[],"_et_theme_builder_marked_as_unused":"","_et_use_on":"","_gallery_link_target":"","_global_colors_info":"","_lh_copy_from_url-original_file":"","_version_history":"","_wp_old_date":["2023-06-25","2023-10-25"],"_wpcode_auto_insert":"","_wpcode_auto_insert_number":"","_wpcode_conditional_logic":[],"_wpcode_conditional_logic_enabled":"","_wpcode_library_id":"","_wpcode_library_version":"","_wpcode_location_extra":"","_wpcode_note":"","_wpcode_priority":"","_wpcode_shortcode_attributes":[],"_wpmf_gallery_custom_image_link":"","ao_post_optimize":[],"author_photo":"268246","doc_url":"","et_enqueued_post_fonts":{"family":{"et-gf-lato":"Lato:100,100italic,300,300italic,regular,italic,700,700italic,900,900italic"},"subset":["latin","latin-ext"],"cache_key":"{\"gph\":0,\"divi\":\"4.24.1\",\"wp\":\"6.6.2\",\"enable_all_character_sets\":\"false\"}"},"rank_math_contentai_score":{"wordCount":"100","linkCount":"0","headingCount":"100","mediaCount":"62.22"},"rank_math_description":"","rank_math_facebook_image":"","rank_math_facebook_image_id":"","rank_math_internal_links_processed":["1","1","1","1","1","1"],"rank_math_og_content_image":{"check":"affdcad0b1601d55e2b82b860bd68126","images":[]},"rank_math_seo_score":"18","rank_math_title":"","version_history":"","wp-smpro-smush-data":[],"wp-smush-animated":"","wpmf_filetype":"","wpmf_order":"","wpmf_size":"","_":"","_bj_lazy_load_skip_post":[],"_divi_filters_post_type":"","_et_dynamic_cached_attributes":{"sticky_position":["top"],"use_custom_gutter":["on"],"fullwidth":["off"],"button_icon":["&#x24;||divi||400"],"social_network":["facebook","twitter","linkedin","youtube","last_fm"],"header_2_font":"|800|||||||","header_2_font_size":"34px","header_2_text_align":"center","animation_intensity_slide":"10%","animation_duration":"800ms","animation_delay":"15ms","animation_intensity_zoom":"15%","animation_intensity_flip":"15%","animation_intensity_fold":"15%","animation_intensity_roll":"15%","animation_direction":"center","animation_style":"none","background_color_gradient_start":"rgba(255,255,255,0)","background_color_gradient_end":"#fafafa","custom_padding":"120px||0px||false|false","background_color_gradient_stops":"rgba(255,255,255,0) 0%|#fafafa 100%","custom_padding_last_edited":"on|desktop","custom_padding_tablet":"||30px||false|false","custom_padding_phone":"60px||||false|false"},"_et_dynamic_cached_shortcodes":["et_pb_post_content","et_pb_contact_field","et_pb_signup_custom_field","et_pb_social_media_follow_network","et_pb_section","et_pb_row","et_pb_column","et_pb_blog","et_pb_blurb","et_pb_button","et_pb_code","et_pb_contact_form","et_pb_post_nav","et_pb_post_title","et_pb_signup","et_pb_social_media_follow","et_pb_text"],"_et_pb_ab_bounce_rate_limit":"","_et_pb_ab_stats_refresh_interval":[],"_et_pb_content_area_background_color":"","_et_pb_dark_text_color":"","_et_pb_light_text_color":"","_et_pb_section_background_color":"","_job_location":"","_job_locations":"","_links_to":"","_links_to_target":"","_product_image_gallery":"","_schema_code":"","_synced_version":"","_wp_attachment_context":"","_wp_attachment_image_alt":[],"_wpie_source_url":"","_yoast_wpseo_content_score":"","_yoast_wpseo_focuskeywords":"","_yoast_wpseo_metadesc":"","_yoast_wpseo_opengraph-image":"","_yst_prominent_words_version":"","inline_featured_image":[],"job_location":[],"job_locations":"","options":"","original-file":"","post_views_count":"","rank_math_analytic_object_id":"","rank_math_canonical_url":"","rank_math_focus_keyword":[],"rank_math_news_sitemap_robots":"","rank_math_primary_category":"0","rank_math_primary_ccategory":"","rank_math_primary_job_locations":"","rank_math_primary_partners_category":"","rank_math_primary_pr_category":"","rank_math_primary_press_release_year":"","rank_math_rich_snippet":"","rank_math_robots":[],"rank_math_schema_Article":[],"rank_math_schema_Organization":[],"rank_math_schema_VideoObject":[],"rank_math_shortcode_schema_s-23675683-fff5-4300-88fe-da8afc8b1bb9":"","rank_math_shortcode_schema_s-307bbc91-c6b1-41aa-950d-c50d435a949c":"","rank_math_shortcode_schema_s-63a052dbc0384":"","rank_math_shortcode_schema_s-63a052dbc039d":"","rank_math_shortcode_schema_s-63a052dbc03a6":"","rank_math_shortcode_schema_s-63a052dbc03aa":"","rank_math_shortcode_schema_s-63a052dbc03b5":"","rank_math_shortcode_schema_s-63a052dbc03ba":"","rank_math_shortcode_schema_s-63a052dbc03bd":"","rank_math_shortcode_schema_s-63b6dd7d53a96":"","rank_math_shortcode_schema_s-63b6dd7d53a9f":"","rank_math_shortcode_schema_s-63b6dd7d53aa2":"","rank_math_shortcode_schema_s-63b6dd7d53aa4":"","rank_math_shortcode_schema_s-63b6dd7d53aa7":"","rank_math_shortcode_schema_s-63b6dd7d53aa9":"","rank_math_shortcode_schema_s-63b6dd7d53aab":"","rank_math_shortcode_schema_s-63b6dd7d53aad":"","rank_math_shortcode_schema_s-63b6dd7d53aaf":"","rank_math_shortcode_schema_s-63c15fcf43311":"","rank_math_shortcode_schema_s-63c15fcf43322":"","rank_math_shortcode_schema_s-63c15fcf43325":"","rank_math_shortcode_schema_s-63c15fcf43327":"","rank_math_shortcode_schema_s-63c15fcf43329":"","rank_math_shortcode_schema_s-63c15fcf4332a":"","rank_math_shortcode_schema_s-63c15fcf4332c":"","rank_math_shortcode_schema_s-63c15fcf4332e":"","rank_math_shortcode_schema_s-63c15fcf43330":"","rank_math_shortcode_schema_s-63f52c5ed40bb":"","rank_math_shortcode_schema_s-6409f40a9b7d5":"","rank_math_shortcode_schema_s-64354a3892419":"","rank_math_shortcode_schema_s-6440158136148":"","rank_math_shortcode_schema_s-6446d2f9353ee":"","rank_math_shortcode_schema_s-6446d2f9353f3":"","rank_math_shortcode_schema_s-6447c0fe4673c":"","rank_math_shortcode_schema_s-64e4d743542d7":"","schema_code":"","smush-complete":"","smush-info":"","smush-stats":[],"synced_version":"","wpmf_remote_video_link":"","_exp":"","_inc":"","_mc4wp_settings":[],"_post-subtitle":"","_pwh_dcfh_contact_email":"","_pwh_dcfh_contact_form_id":"","_pwh_dcfh_form_fields":"","_pwh_dcfh_ip_address":"","_pwh_dcfh_page_id":"","_pwh_dcfh_read_by":"","_pwh_dcfh_referer_url":"","_pwh_dcfh_user_agent":[],"_section1_col1":"","_section1_col2":"","_section1_col3":"","_section1_col4":"","_section2_col1":"","_section2_col2":"","_section2_col3":"","_section2_col4":"","_section2_col5":"","_section2_col6":"","_section3_col1":"","_section3_col2":"","_section3_col3":"","_section3_col4":"","_section3_col5":"","_section3_col6":"","_section4_col1":"","_section4_col2":"","_section4_col3":"","_section4_col4":"","_section4_col5":"","_section4_col6":"","_section5_col1":"","_section5_col2":"","_section5_col3":"","_section5_col4":"","_section5_col5":"","_section5_col6":"","_section6_col1":"","_section6_col2":"","_section6_col3":"","_section6_col4":"","_section6_col5":"","_section6_col6":"","_select_author":"","_test":"","_wp_attachment_backup_sizes":[],"_yoast_wpseo_estimated-reading-time-minutes":[],"_yoast_wpseo_focuskw":[],"_yoast_wpseo_focuskw_text_input":[],"_yoast_wpseo_linkdex":[],"_yoast_wpseo_meta-robots-nofollow":[],"_yoast_wpseo_meta-robots-noindex":[],"_yoast_wpseo_primary_category":[],"_yoast_wpseo_title":[],"_yoast_wpseo_wordproof_timestamp":"","exp":"","inc":"","post-subtitle":[],"rank_math_schema_BlogPosting":[],"section1_col1":"","section1_col2":"","section1_col3":"","section1_col4":"","section2_col1":"","section2_col2":"","section2_col3":"","section2_col4":"","section2_col5":"","section2_col6":"","section3_col1":"","section3_col2":"","section3_col3":"","section3_col4":"","section3_col5":"","section3_col6":"","section4_col1":"","section4_col2":"","section4_col3":"","section4_col4":"","section4_col5":"","section4_col6":"","section5_col1":"","section5_col2":"","section5_col3":"","section5_col4":"","section5_col5":"","section5_col6":"","section6_col1":"","section6_col2":"","section6_col3":"","section6_col4":"","section6_col5":"","section6_col6":"","select_author":"","test":"","footnotes":""},"categories":[1,111,102],"tags":[],"class_list":["post-269159","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog","category-devops","category-kubernetes"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.webscale.com\/wp-json\/wp\/v2\/posts\/269159"}],"collection":[{"href":"https:\/\/www.webscale.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.webscale.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.webscale.com\/wp-json\/wp\/v2\/users\/36"}],"replies":[{"embeddable":true,"href":"https:\/\/www.webscale.com\/wp-json\/wp\/v2\/comments?post=269159"}],"version-history":[{"count":0,"href":"https:\/\/www.webscale.com\/wp-json\/wp\/v2\/posts\/269159\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.webscale.com\/wp-json\/wp\/v2\/media\/269163"}],"wp:attachment":[{"href":"https:\/\/www.webscale.com\/wp-json\/wp\/v2\/media?parent=269159"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.webscale.com\/wp-json\/wp\/v2\/categories?post=269159"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.webscale.com\/wp-json\/wp\/v2\/tags?post=269159"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}