Documentation on the services created by the VMCluster object: Which ones to expose via ingress? How to find out the proper URLs?

[kastl-ars]

Dear maintainers,

we started looking into VictoriaMetrics (finally! :-)) and are very impressed by the smoothness of the deployment and the documentation.

One thing which confuses me and where I think the documentation could benefit from more substance and examples is regarding Kubernetes services and ingress(es) and URLs.

Using VMSingle

When using the VMsingle storage, I found a URL somewhere that I can expose via the VMAuth ingress (or rather disable authentication while exposing it).

spec:
  selectAllByDefault: true
  unauthorizedUserAccessSpec:
    url_prefix:
      - http://<vmsingle-victoriametrics-single>:8428
  ingress:
    class_name: openshift-default
    host: victoriametrics.example.org

(I also figured out which remoteWrite URL I need to set in the VMAgent...)

This works, I can query prometheus metrics and can use this as a datasource in Grafana.
(The targets are not shown, as reported in #1697 but that is just a minor thing and not my topic in this issue)

Using VMCluster

When using VMCluster instead, I had to guess a lot until I figured out that there are two things that can be exposed and do similar things.

• Using http://<vmagent-victoriametrics-agent>:8429 as `url_prefix I can expose the vmagent's GUI.
• Using http://<vmselect-victoriametrics-cluster>:8481/select/0/vmui/ I can expose the vmui

But I have not found documentation on how to expose two services? Do I need two VMAuth objects with different hostnames and url_prefixes (in case I want to disable authentication)?

And the vmui one throws an error when clicking the Explore Prometheus metrics button:

SyntaxError: JSON.parse: unexpected non-whitespace character after JSON data at line 1 column 5 of the JSON data

This seems to come from an error in the vmselect pod:

cannot fetch metric names from vmstorage nodes: cannot search metric names on vmstorage vmstorage-victoriametrics-cluster-<redacted>:8401: cannot execute funcName="searchMetricNames_v3" on vmstorage "100.125.138.99:8401": error when searching for metricIDs: the number of matching timeseries exceeds 30000; either narrow down the search or increase -search.max* command-line flag values (the most likely limit is -search.maxUniqueTimeseries)

This is a little surprising, as there is next to no workload being scraped. And this worked out of the box without issues using the VMSingle storage (in the same cluster). Did I pick the right URLs (or did I put the wrong URL somewhere?

Also, there are some kubernetes services called ...-internal which I did not find documentation about.

It could really be that I just misread or did not find the proper parts of the documentation, so I would gladly accept a link to the proper docs...

Thanks in advance!

Kind Regards,
Johannes

[AndrewChubatiuk]

Hi @kastl-ars, created PR with updates for the docs to cover this case.

---

Services created by VMCluster

For a VMCluster named <name> in namespace <namespace>, the operator creates:

Service name	Type	Port	Purpose
vminsert-<name>	ClusterIP	8480	Metrics ingestion (write path)
vmselect-<name>	Headless	8481	Metrics querying (read path)
vmstorage-<name>	Headless	8482	vmstorage admin HTTP API (not for direct user access)

The -internal services (vminsertinternal-<name>, vmselectinternal-<name>, vmclusterlb-<name>) only appear when requestsLoadBalancer is enabled — they are created internally for the VMAuth-based load balancer and are not intended for direct use.

Common URLs

VictoriaMetrics cluster uses a tenant model. For single-tenant setups, the default tenant is 0:

Use case	URL
Prometheus remote write (VMAgent)	http://vminsert-<name>.<namespace>.svc:8480/insert/0/prometheus/api/v1/write
PromQL / MetricsQL query API	http://vmselect-<name>.<namespace>.svc:8481/select/0/prometheus
Grafana datasource URL	http://vmselect-<name>.<namespace>.svc:8481/select/0/prometheus
VMUI (built-in web UI)	http://vmselect-<name>.<namespace>.svc:8481/select/0/vmui/

The URL pattern you found (http://<vmselect-...>:8481/select/0/vmui/) is correct.

Exposing via VMAuth / Ingress

Because the write and read paths are separate services, you expose them through a single VMAuth ingress by routing with VMUser objects. You do not need two separate VMAuth objects with different hostnames — path-based routing is sufficient:

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAuth
metadata:
  name: cluster-proxy
  namespace: default
spec:
  selectAllByDefault: true
  ingress:
    class_name: openshift-default   # your ingress class
    host: victoriametrics.example.org
  # public read access (no auth required):
  unauthorizedUserAccessSpec:
    targetRefs:
      - crd:
          kind: VMCluster/vmselect
          name: <cluster-name>
          namespace: default
        target_path_suffix: "/select/0"
        paths:
          - "/prometheus/.*"
          - "/vmui.*"
---
# write access for VMAgent:
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMUser
metadata:
  name: cluster-writer
  namespace: default
spec:
  username: writer
  generatePassword: true
  targetRefs:
    - crd:
        kind: VMCluster/vminsert
        name: <cluster-name>
        namespace: default
      target_path_suffix: "/insert/0"
      paths:
        - "/prometheus/.*"

The error you saw is unrelated to the URL — you have the right URL. This is a safety limit on vmselect that prevents queries from listing all metric names when there are more than 30 000 unique time series. You can raise it via extraArgs on vmselect:

spec:
  vmselect:
    extraArgs:
      search.maxUniqueTimeseries: "100000"