{"id":269212,"date":"2023-07-31T13:01:23","date_gmt":"2023-07-31T18:01:23","guid":{"rendered":"https:\/\/www.webscale.com\/?p=269190"},"modified":"2023-12-29T15:30:58","modified_gmt":"2023-12-29T20:30:58","slug":"prometheus-querying","status":"publish","type":"post","link":"https:\/\/www.webscale.com\/blog\/prometheus-querying\/","title":{"rendered":"Prometheus Querying &#8211; Breaking Down PromQL"},"content":{"rendered":"<p>Prometheus has its own language specifically dedicated to queries called\u00a0<a href=\"https:\/\/prometheus.io\/docs\/prometheus\/latest\/querying\/basics\/\" target=\"_blank\" rel=\"noopener\">PromQL<\/a>. It is a powerful functional expression language, which lets you filter with Prometheus\u2019 multi-dimensional time-series labels. The result of each expression can be shown either as a graph, viewed as tabular data within Prometheus\u2019 own expression browser, or consumed via external systems via the\u00a0<a href=\"https:\/\/prometheus.io\/docs\/prometheus\/latest\/querying\/api\/\" target=\"_blank\" rel=\"noopener\">HTTP API<\/a>.<\/p>\n<p>PromQL can be a difficult language to understand, particularly if you are faced with an empty input field and are having to come up with the formation of queries on your own. This article is a primer dedicated to the basics of how to run Prometheus queries.<\/p>\n<p>You will need only three tools:<\/p>\n<ul>\n<li>Prometheus server port forwarded from the local connection<\/li>\n<li>Simple cURL<\/li>\n<li>A data visualization and monitoring tool, either within Prometheus or an external one, such as Grafana<\/li>\n<\/ul>\n<p>Through query building, you will end up with a graph per CPU by the deployment.<\/p>\n<h2 id=\"prometheus-querying\">Prometheus Querying<\/h2>\n<p>The core part of any query in PromQL are the metric names of a time-series. Indeed, all Prometheus metrics are time based data. There are four parts to every metric. Taking the\u00a0<code>varnish_main_client_req<\/code>\u00a0metric as an example:<\/p>\n<div><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-269214\" src=\"https:\/\/www.webscale.com\/wp-content\/uploads\/2023\/10\/varnish-main-client-req-metric-1024x119.png\" alt=\"\" width=\"1024\" height=\"119\" \/><\/div>\n<p>The parts are:<\/p>\n<ol>\n<li style=\"list-style-type: none\">\n<ol>\n<li><strong>Metric_name<\/strong>\u00a0(e.g.\u00a0<code>varnish_main_client_req<\/code>)<\/li>\n<li>One or more\u00a0<strong>labels<\/strong>, which are simply key-value pairs that distinguish each metric with the same name (e.g.\u00a0<code>namespace=\"section-b4a199920b24b\"<\/code>). Each metric will have at least a\u00a0<code>job<\/code>\u00a0label, which corresponds to the scrape config in the prometheus config.<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<ol>\n<li>The\u00a0<strong>value<\/strong>, which is a\u00a0<a href=\"https:\/\/golang.org\/pkg\/builtin\/#float64\" target=\"_blank\" rel=\"noopener\">float64<\/a>. When querying in the Prometheus console, the value shown is the value for the most recent timestamp collected.<\/li>\n<li>The\u00a0<strong>timestamp<\/strong>, which has millisecond precision. The timestamp doesn\u2019t appear in the query console, but if you switch to the Graph tab within Prometheus, you can see the values for each timestamp in the chart.<\/li>\n<\/ol>\n<p>Each distinct metric_name &amp; label combination is called a\u00a0<strong>time-series<\/strong>\u00a0(often just called a series in the documentation). If each series only has a single value for each timestamp, as in the above example, the collection of series returned from a query is called an\u00a0<strong>instant-vector<\/strong>. If each series has multiple values, it is referred to as a\u00a0<strong>range-vector<\/strong>. These are generated by appending a time selector to the instant-vector in square brackets (e.g. [5m] for five minutes). The instant vector and range vector are two of four types of expression language; the final two are scalar, a simple numeric floating point value, and string, a simple string value. See Range Selectors below for further information on this.<\/p>\n<p>All of these metrics are scraped from\u00a0<a href=\"https:\/\/prometheus.io\/docs\/instrumenting\/exporters\/\" target=\"_blank\" rel=\"noopener\">exporters<\/a>. Prometheus scrapes these metrics at regular intervals. The setting for when the intervals should occur is specified by the scrape_interval in the prometheus.yaml config. Most scrape intervals are 30s. This means that every 30s, there will be a new data point with a new timestamp. The value may or may not have changed, but at every scrape_interval, there will be a new datapoint.<\/p>\n<p>There are four types of metrics:<\/p>\n<ul>\n<li><strong>Counters<\/strong>\u00a0&#8211; A cumulative, monotonic metric. Counters allow the value to either go up, stay the same or be reset to 0 (on a process restart).\u00a0<code>varnish_main_client_req<\/code>\u00a0is an example of this, which provides the total number of HTTP requests that the varnish instance has handled in its life.<\/li>\n<li><strong>Gauges<\/strong>\u00a0&#8211; A non-monotonic metric. Gauges can go either up or down, giving the current value at any given point in time. An example is\u00a0<code>node_memory_utilisation<\/code>, which provides the current percentage of memory used on each node.<\/li>\n<li><strong>Histogram<\/strong>\u00a0&#8211; This creates multiple series for each metric name. Sampled values are put into buckets. Sum &amp; count metrics are also generated for each sample.<\/li>\n<li><strong>Summary<\/strong>\u00a0&#8211; The summary is similar to histogram in that it takes samples and creates multiple metrics, including sum &amp; count. However, it creates quantiles (i.e. 50th percentile, 90th percentile, etc.) instead of buckets.<\/li>\n<\/ul>\n<p>We\u2019re going to deal with counters for this analysis, as it\u2019s the most common metric type.<\/p>\n<h2 id=\"query-structure\">Query Structure<\/h2>\n<p>The structure of a basic Prometheus query looks very much like a metric. You start with a metric name. If you just query\u00a0<code>varnish_main_client_req<\/code>, every one of those metrics for every varnish pod in every namespace will get returned. If you do this in Grafana, you risk crashing the browser tab as it tries to render so many data points simultaneously.<\/p>\n<p>Next, you can filter the query using labels. Label filters support four operators:<\/p>\n<ul>\n<li><code>=<\/code>\u00a0equal<\/li>\n<li><code>!=<\/code>\u00a0not-equal<\/li>\n<li><code>=~<\/code>\u00a0matches regex<\/li>\n<li><code>!~<\/code>\u00a0doesn\u2019t match regex<\/li>\n<\/ul>\n<p>Label filters go inside the {} after the metric name, so an equality match looks like:<\/p>\n<pre><code>varnish_main_client_req{namespace=\"section-9469f9cc28d8d\"}<\/code><\/pre>\n<p>which will return only\u00a0<code>varnish_main_client_req<\/code>\u00a0metrics with that exact namespace.<\/p>\n<p>Regex matches use the\u00a0<a href=\"https:\/\/github.com\/google\/re2\/wiki\/Syntax\" target=\"_blank\" rel=\"noopener\">RE2 syntax<\/a>. If you\u2019re familiar with\u00a0<a href=\"https:\/\/www.pcre.org\/\" target=\"_blank\" rel=\"noopener\">PCRE<\/a>, it will look much the same, but it doesn\u2019t support backreferences (which really shouldn\u2019t matter here anyway).<\/p>\n<p>You can also use multiple label filters, separated by a comma. Multiple label filters are an \u201cAND\u201d query, so in order to be returned, a metric must match all the label filters.<br \/>\nFor instance,<\/p>\n<pre><code>varnish_main_client_req{namespace=~\".*3.*\",namespace!~\".*env4.*\"}<\/code><\/pre>\n<p>will return all\u00a0<code>varnish_main_client_req<\/code>\u00a0metrics with a 3 in their namespace that don\u2019t also contain\u00a0<em>env4<\/em>.<\/p>\n<h2 id=\"range-selectors\">Range Selectors<\/h2>\n<p>By appending a\u00a0<a href=\"https:\/\/prometheus.io\/docs\/prometheus\/latest\/querying\/basics\/#range-vector-selectors\" target=\"_blank\" rel=\"noopener\">range duration<\/a>\u00a0to a query, we get multiple values for each timestamp. This is referred to as a range-vector. The values for each timestamp will be the values recorded in the time series back in time, taken from the timestamp for the length of time given in the range duration.<\/p>\n<p>As an example, take\u00a0<code>varnish_main_client_req{namespace=\"section-9469f9cc28d8d\"}<\/code>. If we add a [1m] range selector we now get this:<\/p>\n<div><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-269216\" src=\"https:\/\/www.webscale.com\/wp-content\/uploads\/2023\/10\/varnish-main-client-req-metric-1m-1024x107.png\" alt=\"\" width=\"1024\" height=\"107\" \/><\/div>\n<p>We get two values for each series because the varnish scrape config specifies that it has a 30 second interval, so if you look at the timestamps after the @ symbol in the value, you can see that they are exactly 30 seconds apart. If you graphed these series without the range selector and inspected the value of the lines at those timestamps, it would show these values.<\/p>\n<p>Now, range-vectors can\u2019t be graphed because they have multiple values for each timestamp. If you select the Graph tab in the Prometheus web UI on a range-vector, you\u2019ll see this message:<\/p>\n<p><code>Error executing query: invalid expression type \"range vector\" for range query, must be Scalar or instant Vector<\/code><\/p>\n<p>A range-vector is typically generated in order to then apply a function to it to get an instant-vector, which can be graphed (only instant vectors can be graphed). Prometheus has\u00a0<a href=\"https:\/\/prometheus.io\/docs\/prometheus\/latest\/querying\/functions\" target=\"_blank\" rel=\"noopener\">many functions<\/a>\u00a0for both instant and range vectors. The more commonly used functions for working with range-vectors are:<\/p>\n<ul>\n<li><code>rate()<\/code>\u00a0&#8211; calculates the per-second average rate of increase of the time series in the range vector over the whole range.<\/li>\n<li><code>irate()<\/code>\u00a0&#8211; calculates the per-second average rate of increase of the time series in the range vector\u00a0<strong>using only the last two data points in the range<\/strong>.<\/li>\n<li><code>increase()<\/code>\u00a0&#8211; calculates the increase in the time series per the time range selected. It\u2019s basically rate multiplied by the number of seconds in the time range selector.<\/li>\n<\/ul>\n<p>Your selection of range duration will determine how granular your chart is. A [1m] duration, for instance, will give a very spiky chart, making it difficult to visualize a trend, looking something like this:<\/p>\n<div><\/div>\n<div><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-269217\" src=\"https:\/\/www.webscale.com\/wp-content\/uploads\/2023\/10\/promql-spiky-chart-1024x562.png\" alt=\"\" width=\"1024\" height=\"562\" \/><\/div>\n<p>For a one hour view, [5m] would show a decent view:<\/p>\n<div><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-269218\" src=\"https:\/\/www.webscale.com\/wp-content\/uploads\/2023\/10\/promql-5m-graph-1024x544.png\" alt=\"\" width=\"1024\" height=\"544\" \/><\/div>\n<p>For longer time-spans, you may want to set a longer range duration to help smooth out spikes and achieve more of a long-term trend view. Compare the three day view with a [5m] duration to a [30m] duration:<\/p>\n<div><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-269219\" src=\"https:\/\/www.webscale.com\/wp-content\/uploads\/2023\/10\/promql-5m-comparison-graph-1024x543.png\" alt=\"\" width=\"1024\" height=\"543\" \/><\/div>\n<div><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-269220\" src=\"https:\/\/www.webscale.com\/wp-content\/uploads\/2023\/10\/promql-30m-comparison-graph-1024x549.png\" alt=\"\" width=\"1024\" height=\"549\" \/><\/div>\n<h2 id=\"joining-series\">Joining Series<\/h2>\n<p>Technically, Prometheus doesn\u2019t have the concept of joining series like SQL, for example, has. However, series can be combined in Prometheus by using an\u00a0<a href=\"https:\/\/prometheus.io\/docs\/prometheus\/latest\/querying\/operators\/\" target=\"_blank\" rel=\"noopener\">operator on them<\/a>. It has the usual arithmetic, comparison and logical operators that can be applied to multiple series or to scalar values.<\/p>\n<p>A caution: if an operator is applied to two instant-vectors, it will only apply to matching series. A series is considered to match if and only if it has exactly the same set of labels. Using these operators on series achieves a one-to-one matching when each series from the left side of the expression exactly matches one series on the right side.<\/p>\n<p>So, taking the two series from the previous examples as separate instant-vectors:<\/p>\n<pre><code>varnish_main_client_req{endpoint=\"metrics\",instance=\"10.244.24.68:9131\",job=\"varnish\",namespace=\"section-9469f9cc28d8d\",pod=\"varnish-786d4648bd-lrlrc\",service=\"p8s-varnish\"}<\/code><\/pre>\n<p>&amp;<\/p>\n<pre><code>varnish_main_client_req{endpoint=\"metrics\",instance=\"10.244.48.66:9131\",job=\"varnish\",namespace=\"section-9469f9cc28d8d\",pod=\"varnish-786d4648bd-rfnjb\",service=\"p8s-varnish\"}<\/code><\/pre>\n<p>If we apply an addition operator to these two to try and get a total number of requests in that namespace, nothing will be returned.<\/p>\n<div><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-269221\" src=\"https:\/\/www.webscale.com\/wp-content\/uploads\/2023\/10\/promql-no-data-1024x191.png\" alt=\"\" width=\"1024\" height=\"191\" \/><\/div>\n<p>This is because there are no series returned that have exactly matching labels. However, if we use the\u00a0<code>on<\/code>\u00a0keyword to specify that we only want to match on the\u00a0<code>namespace<\/code>\u00a0label, we get:<\/p>\n<div><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-269222\" src=\"https:\/\/www.webscale.com\/wp-content\/uploads\/2023\/10\/promql-namespace-match-1024x188.png\" alt=\"\" width=\"1024\" height=\"188\" \/><\/div>\n<p>Note that the new instant-vector contains a single series with only the label(s) specified in the\u00a0<code>on<\/code>\u00a0keyword.<\/p>\n<p>The\u00a0<code>ignoring<\/code>\u00a0keyword can also be used as an inverse of that to specify which labels should be ignored when trying to match.<\/p>\n<div><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-269223\" src=\"https:\/\/www.webscale.com\/wp-content\/uploads\/2023\/10\/promql-ignoring-keyword-1024x196.png\" alt=\"\" width=\"1024\" height=\"196\" \/><\/div>\n<p>In this case the returned instant-vector contains a single series with all matching labels left after removing the labels in the\u00a0<code>ignoring<\/code>\u00a0set.<\/p>\n<p>It is worth pointing out here that Prometheus also has a number of\u00a0<a href=\"https:\/\/prometheus.io\/docs\/prometheus\/latest\/querying\/operators\/#aggregation-operators\" target=\"_blank\" rel=\"noopener\">aggregation operators<\/a>. So if we really wanted to get the total number of client requests in a namespace, we would never actually do this because the pod names would change over time. The\u00a0<code>sum<\/code>\u00a0operator is much simpler:<\/p>\n<p><code>sum(varnish_main_client_req{namespace=\"section-9469f9cc28d8d\"}) by (namespace)<\/code><\/p>\n<h2 id=\"combining-instant-vectors-with-scalars\">Combining Instant-Vectors with Scalars<\/h2>\n<p>You can also combine an instant-vector with a scalar value. Each series in the instant-vector has the value applied with the operator. For example:<\/p>\n<p><code>varnish_main_client_req{namespace=\"section-9469f9cc28d8d\"} * 10<\/code><\/p>\n<p>results in each value for each series in the instant-vector being multiplied by ten.<\/p>\n<div><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-269224\" src=\"https:\/\/www.webscale.com\/wp-content\/uploads\/2023\/10\/promql-instant-vector-scalar-1024x86.png\" alt=\"\" width=\"1024\" height=\"86\" \/><\/div>\n<p>This can be useful for calculating ratios &amp; percentages.<\/p>\n<h2 id=\"kubernetes-reference-series\">Kubernetes Reference Series<\/h2>\n<p>What if we wanted to know how many requests were being made against each node in our Kubernetes cluster? We can see this in the\u00a0<code>varnish_main_client_req<\/code>\u00a0series by looking at the\u00a0<code>instance<\/code>\u00a0label (e.g.\u00a0<code>instance=\"10.244.48.66:9131\"<\/code>). However, this is not particularly helpful as the IP address given there is only the internal IP address of the pod as addressed by the service the Prometheus operator has used to identify which endpoints to scrape. The port is t that which the\u00a0<a href=\"https:\/\/github.com\/jonnenauha\/prometheus_varnish_exporter\" target=\"_blank\" rel=\"noopener\">varnishstat promethues exporter<\/a>\u00a0publishes its metrics endpoint to by default.<\/p>\n<p>To address this, the kube-state-metrics and node-exporter Prometheus exporters publish a number of series that essentially exist to provide reference labels. These are series like\u00a0<code>kube_pod_info<\/code>\u00a0&amp;\u00a0<code>node_uname_info<\/code>.<\/p>\n<p>There is a\u00a0<code>kube_pod_info<\/code>\u00a0series for every single pod in the cluster. If we filter it down to the two pods we are interested in knowing more about, we can see what kind of information it provides:<\/p>\n<div><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-269225\" src=\"https:\/\/www.webscale.com\/wp-content\/uploads\/2023\/10\/promql-kube_pod_info-1024x289.png\" alt=\"\" width=\"1024\" height=\"289\" \/><\/div>\n<p>From this, we can see the name of the node, the IP address of the nodes and which ReplicaSet created the pods.<\/p>\n<p>Also note the value. The value for these reference series is always 1. The reason for this is so that you can join them with other series simply by multiplying without having to change the value of the original series, and still gain access to the new set of labels.<\/p>\n<p>So, to get our\u00a0<code>varnish_main_client_req<\/code>\u00a0series with the\u00a0<code>node<\/code>\u00a0label on them, we can do this :<\/p>\n<p><code>varnish_main_client_req{namespace=\"section-9469f9cc28d8d\"} * on (pod) kube_pod_info<\/code><\/p>\n<p>This returns us:<\/p>\n<div><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-269226\" src=\"https:\/\/www.webscale.com\/wp-content\/uploads\/2023\/10\/promql-node-label-1024x86.png\" alt=\"\" width=\"1024\" height=\"86\" \/><\/div>\n<p>Unfortunately because we needed to use the on keyword to match, that is also the only label we get back. Ignoring non matching labels won\u2019t help here because pod is the only matching label. To fix this, we use the\u00a0<code>group_left<\/code>\u00a0or\u00a0<code>group_right<\/code>\u00a0keywords.<\/p>\n<p>These keywords convert the match into a many-to-one or one-to-many matching respectively. The left and right indicate the side that has the higher cardinality. So a group_left means that multiple series on the left side can match a single series on the right. The result of this is that the returned instant-vector contains all of the labels from the side with the higher cardinality, even if they don\u2019t match any label on the right.<\/p>\n<p>So\u00a0<code>varnish_main_client_req{namespace=\"section-9469f9cc28d8d\"} * on (pod) group_left() kube_pod_info<\/code>\u00a0gives:<\/p>\n<div><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-269227\" src=\"https:\/\/www.webscale.com\/wp-content\/uploads\/2023\/10\/promql-group-left-right-1024x79.png\" alt=\"\" width=\"1024\" height=\"79\" \/><\/div>\n<p>This gives us back all the labels for the\u00a0<code>varnish_main_client_req<\/code>\u00a0series, however we still don\u2019t have the\u00a0<code>node<\/code>\u00a0label. To solve this, the\u00a0<code>group_left<\/code>\u00a0and\u00a0<code>group_right<\/code>\u00a0keywords allow a label list to be passed in. The labels provided will be included from the matching lower cardinality side of the operation. So by doing:<\/p>\n<p><code>varnish_main_client_req{namespace=\"section-9469f9cc28d8d\"} * on (pod) group_left(node) kube_pod_info<\/code><\/p>\n<p>We can get all the information required:<\/p>\n<div><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-269228\" src=\"https:\/\/www.webscale.com\/wp-content\/uploads\/2023\/10\/promql-node-group-left-right-1024x80.png\" alt=\"\" width=\"1024\" height=\"80\" \/><\/div>\n<p>We can then take this over to Grafana to make a dashboard and chart, add this data to a graph panel, and clearly view it all.<\/p>\n<div><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-269212\" src=\"https:\/\/www.webscale.com\/wp-content\/uploads\/2023\/10\/promql-grafana-1024x676.png\" alt=\"\" width=\"1024\" height=\"676\" \/><\/div>\n<h2 id=\"avoiding-overloads-and-slow-queries\">Avoiding Overloads and Slow Queries<\/h2>\n<p>Sometimes graphing a query might overload the server or browser, or lead to a time out because the amount of data is too large. When constructing queries over unknown data, it is better to begin building the query in the tabular view of Prometheus\u2019 expression browser until you arrive at a reasonable result set (i.e. hundreds as opposed to thousands of time series). Switch to graph mode only once you have sufficiently aggregated or filtered your data. If the expression continues to take too long to graph ad-hoc, you can pre-cord it using a recording rule. This is particularly relevant to PromQL when a bare metric name selector such as\u00a0<code>api_http_requests_total<\/code>\u00a0can easily expand to thousands of time series each with a different label. Also, expressions that aggregate over multiple time series will generate load on the server even when the output is only a small amount of time series.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Prometheus has its own language specifically dedicated to queries called\u00a0PromQL. It is a powerful functional expression language, which lets you filter with Prometheus\u2019 multi-dimensional time-series labels. The result of each expression can be shown either as a graph, viewed as tabular data within Prometheus\u2019 own expression browser, or consumed via external systems via the\u00a0HTTP API. [&hellip;]<\/p>\n","protected":false},"author":36,"featured_media":269280,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","rank_math_lock_modified_date":false,"_aioseo_description":"","_aioseo_keywords":"","_aioseo_og_article_section":"","_aioseo_og_article_tags":"","_aioseo_og_description":"","_aioseo_og_title":"","_aioseo_title":"","_aioseo_twitter_description":"","_aioseo_twitter_title":"","_author_photo":"field_6513304084a08","_doc_url":"","_dp_original":"","_et_autogenerated_title":"","_et_body_layout_enabled":"","_et_body_layout_id":"","_et_builder_dynamic_assets_loading_attr_threshold":"2","_et_builder_module_features_cache":null,"_et_builder_version":"","_et_default":"","_et_enabled":"","_et_footer_layout_enabled":"","_et_footer_layout_id":"","_et_header_layout_enabled":"","_et_header_layout_id":"","_et_pb_ab_current_shortcode":"","_et_pb_ab_subjects":"","_et_pb_built_for_post_type":"","_et_pb_custom_css":"","_et_pb_enable_shortcode_tracking":"","_et_pb_excluded_global_options":"","_et_pb_first_image":"","_et_pb_gutter_width":"","_et_pb_module_type":"","_et_pb_page_layout":"et_right_sidebar","_et_pb_page_z_index":"","_et_pb_post_hide_nav":"default","_et_pb_row_layout":"","_et_pb_show_page_creation":"","_et_pb_show_title":"on","_et_pb_side_nav":"off","_et_pb_static_css_file":"","_et_pb_truncate_post":"","_et_pb_truncate_post_date":"","_et_post_bg_color":"#ffffff","_et_post_bg_layout":"light","_et_template":[],"_et_theme_builder_marked_as_unused":"","_et_use_on":"","_gallery_link_target":"","_global_colors_info":"","_lh_copy_from_url-original_file":"","_version_history":"","_wp_old_date":["2023-10-26"],"_wpcode_auto_insert":"","_wpcode_auto_insert_number":"","_wpcode_conditional_logic":[],"_wpcode_conditional_logic_enabled":"","_wpcode_library_id":"","_wpcode_library_version":"","_wpcode_location_extra":"","_wpcode_note":"","_wpcode_priority":"","_wpcode_shortcode_attributes":[],"_wpmf_gallery_custom_image_link":"","ao_post_optimize":[],"author_photo":"268246","doc_url":"","et_enqueued_post_fonts":{"family":{"et-gf-lato":"Lato:100,100italic,300,300italic,regular,italic,700,700italic,900,900italic"},"subset":["latin","latin-ext"],"cache_key":"{\"gph\":0,\"divi\":\"4.24.1\",\"wp\":\"6.6.2\",\"enable_all_character_sets\":\"false\"}"},"rank_math_contentai_score":{"wordCount":"100","linkCount":"0","headingCount":"100","mediaCount":"62.22"},"rank_math_description":"","rank_math_facebook_image":"","rank_math_facebook_image_id":"","rank_math_internal_links_processed":["1","1"],"rank_math_og_content_image":{"check":"755af227b45bc0a27954f2a566b28d98","images":[]},"rank_math_seo_score":"27","rank_math_title":"","version_history":"","wp-smpro-smush-data":[],"wp-smush-animated":"","wpmf_filetype":"","wpmf_order":"","wpmf_size":"","_":"","_bj_lazy_load_skip_post":[],"_divi_filters_post_type":"","_et_dynamic_cached_attributes":{"sticky_position":["top"],"use_custom_gutter":["on"],"fullwidth":["off"],"button_icon":["&#x24;||divi||400"],"social_network":["facebook","twitter","linkedin","youtube","last_fm"],"header_2_font":"|800|||||||","header_2_font_size":"34px","header_2_text_align":"center","animation_intensity_slide":"10%","animation_duration":"800ms","animation_delay":"15ms","animation_intensity_zoom":"15%","animation_intensity_flip":"15%","animation_intensity_fold":"15%","animation_intensity_roll":"15%","animation_direction":"center","animation_style":"none","background_color_gradient_start":"rgba(255,255,255,0)","background_color_gradient_end":"#fafafa","custom_padding":"120px||0px||false|false","background_color_gradient_stops":"rgba(255,255,255,0) 0%|#fafafa 100%","custom_padding_last_edited":"on|desktop","custom_padding_tablet":"||30px||false|false","custom_padding_phone":"60px||||false|false"},"_et_dynamic_cached_shortcodes":["et_pb_post_content","et_pb_contact_field","et_pb_signup_custom_field","et_pb_social_media_follow_network","et_pb_section","et_pb_row","et_pb_column","et_pb_blog","et_pb_blurb","et_pb_button","et_pb_code","et_pb_contact_form","et_pb_post_nav","et_pb_post_title","et_pb_signup","et_pb_social_media_follow","et_pb_text"],"_et_pb_ab_bounce_rate_limit":"","_et_pb_ab_stats_refresh_interval":[],"_et_pb_content_area_background_color":"","_et_pb_dark_text_color":"","_et_pb_light_text_color":"","_et_pb_section_background_color":"","_job_location":"","_job_locations":"","_links_to":"","_links_to_target":"","_product_image_gallery":"","_schema_code":"","_synced_version":"","_wp_attachment_context":"","_wp_attachment_image_alt":[],"_wpie_source_url":"","_yoast_wpseo_content_score":"","_yoast_wpseo_focuskeywords":"","_yoast_wpseo_metadesc":"","_yoast_wpseo_opengraph-image":"","_yst_prominent_words_version":"","inline_featured_image":[],"job_location":[],"job_locations":"","options":"","original-file":"","post_views_count":"","rank_math_analytic_object_id":"","rank_math_canonical_url":"","rank_math_focus_keyword":[],"rank_math_news_sitemap_robots":"","rank_math_primary_category":"0","rank_math_primary_ccategory":"","rank_math_primary_job_locations":"","rank_math_primary_partners_category":"","rank_math_primary_pr_category":"","rank_math_primary_press_release_year":"","rank_math_rich_snippet":"","rank_math_robots":[],"rank_math_schema_Article":[],"rank_math_schema_Organization":[],"rank_math_schema_VideoObject":[],"rank_math_shortcode_schema_s-23675683-fff5-4300-88fe-da8afc8b1bb9":"","rank_math_shortcode_schema_s-307bbc91-c6b1-41aa-950d-c50d435a949c":"","rank_math_shortcode_schema_s-63a052dbc0384":"","rank_math_shortcode_schema_s-63a052dbc039d":"","rank_math_shortcode_schema_s-63a052dbc03a6":"","rank_math_shortcode_schema_s-63a052dbc03aa":"","rank_math_shortcode_schema_s-63a052dbc03b5":"","rank_math_shortcode_schema_s-63a052dbc03ba":"","rank_math_shortcode_schema_s-63a052dbc03bd":"","rank_math_shortcode_schema_s-63b6dd7d53a96":"","rank_math_shortcode_schema_s-63b6dd7d53a9f":"","rank_math_shortcode_schema_s-63b6dd7d53aa2":"","rank_math_shortcode_schema_s-63b6dd7d53aa4":"","rank_math_shortcode_schema_s-63b6dd7d53aa7":"","rank_math_shortcode_schema_s-63b6dd7d53aa9":"","rank_math_shortcode_schema_s-63b6dd7d53aab":"","rank_math_shortcode_schema_s-63b6dd7d53aad":"","rank_math_shortcode_schema_s-63b6dd7d53aaf":"","rank_math_shortcode_schema_s-63c15fcf43311":"","rank_math_shortcode_schema_s-63c15fcf43322":"","rank_math_shortcode_schema_s-63c15fcf43325":"","rank_math_shortcode_schema_s-63c15fcf43327":"","rank_math_shortcode_schema_s-63c15fcf43329":"","rank_math_shortcode_schema_s-63c15fcf4332a":"","rank_math_shortcode_schema_s-63c15fcf4332c":"","rank_math_shortcode_schema_s-63c15fcf4332e":"","rank_math_shortcode_schema_s-63c15fcf43330":"","rank_math_shortcode_schema_s-63f52c5ed40bb":"","rank_math_shortcode_schema_s-6409f40a9b7d5":"","rank_math_shortcode_schema_s-64354a3892419":"","rank_math_shortcode_schema_s-6440158136148":"","rank_math_shortcode_schema_s-6446d2f9353ee":"","rank_math_shortcode_schema_s-6446d2f9353f3":"","rank_math_shortcode_schema_s-6447c0fe4673c":"","rank_math_shortcode_schema_s-64e4d743542d7":"","schema_code":"","smush-complete":"","smush-info":"","smush-stats":[],"synced_version":"","wpmf_remote_video_link":"","_exp":"","_inc":"","_mc4wp_settings":[],"_post-subtitle":"","_pwh_dcfh_contact_email":"","_pwh_dcfh_contact_form_id":"","_pwh_dcfh_form_fields":"","_pwh_dcfh_ip_address":"","_pwh_dcfh_page_id":"","_pwh_dcfh_read_by":"","_pwh_dcfh_referer_url":"","_pwh_dcfh_user_agent":[],"_section1_col1":"","_section1_col2":"","_section1_col3":"","_section1_col4":"","_section2_col1":"","_section2_col2":"","_section2_col3":"","_section2_col4":"","_section2_col5":"","_section2_col6":"","_section3_col1":"","_section3_col2":"","_section3_col3":"","_section3_col4":"","_section3_col5":"","_section3_col6":"","_section4_col1":"","_section4_col2":"","_section4_col3":"","_section4_col4":"","_section4_col5":"","_section4_col6":"","_section5_col1":"","_section5_col2":"","_section5_col3":"","_section5_col4":"","_section5_col5":"","_section5_col6":"","_section6_col1":"","_section6_col2":"","_section6_col3":"","_section6_col4":"","_section6_col5":"","_section6_col6":"","_select_author":"","_test":"","_wp_attachment_backup_sizes":[],"_yoast_wpseo_estimated-reading-time-minutes":[],"_yoast_wpseo_focuskw":[],"_yoast_wpseo_focuskw_text_input":[],"_yoast_wpseo_linkdex":[],"_yoast_wpseo_meta-robots-nofollow":[],"_yoast_wpseo_meta-robots-noindex":[],"_yoast_wpseo_primary_category":[],"_yoast_wpseo_title":[],"_yoast_wpseo_wordproof_timestamp":"","exp":"","inc":"","post-subtitle":[],"rank_math_schema_BlogPosting":[],"section1_col1":"","section1_col2":"","section1_col3":"","section1_col4":"","section2_col1":"","section2_col2":"","section2_col3":"","section2_col4":"","section2_col5":"","section2_col6":"","section3_col1":"","section3_col2":"","section3_col3":"","section3_col4":"","section3_col5":"","section3_col6":"","section4_col1":"","section4_col2":"","section4_col3":"","section4_col4":"","section4_col5":"","section4_col6":"","section5_col1":"","section5_col2":"","section5_col3":"","section5_col4":"","section5_col5":"","section5_col6":"","section6_col1":"","section6_col2":"","section6_col3":"","section6_col4":"","section6_col5":"","section6_col6":"","select_author":"","test":"","footnotes":""},"categories":[1,111],"tags":[],"class_list":["post-269212","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog","category-devops"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.webscale.com\/wp-json\/wp\/v2\/posts\/269212"}],"collection":[{"href":"https:\/\/www.webscale.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.webscale.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.webscale.com\/wp-json\/wp\/v2\/users\/36"}],"replies":[{"embeddable":true,"href":"https:\/\/www.webscale.com\/wp-json\/wp\/v2\/comments?post=269212"}],"version-history":[{"count":0,"href":"https:\/\/www.webscale.com\/wp-json\/wp\/v2\/posts\/269212\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.webscale.com\/wp-json\/wp\/v2\/media\/269280"}],"wp:attachment":[{"href":"https:\/\/www.webscale.com\/wp-json\/wp\/v2\/media?parent=269212"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.webscale.com\/wp-json\/wp\/v2\/categories?post=269212"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.webscale.com\/wp-json\/wp\/v2\/tags?post=269212"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}