75+ Splunk Interview Questions for DevOps Engineers (2027)

76 Splunk interview questions for DevOps engineers: forwarders, HEC, props.conf, SPL, Kubernetes via OpenTelemetry, clustering and scenarios.

Sep 13, 2025 - 17:43
Nov 11, 2026 - 19:27
 0  79
75+ Splunk Interview Questions for DevOps Engineers (2027)

Splunk interview questions for DevOps engineers focus on how data gets in (universal forwarders, the HTTP Event Collector, OpenTelemetry), how it is stored (indexes, buckets, indexers), and how you search it with SPL. This page has 76 questions with direct answers, grouped for freshers, engineers with 2 to 5 years of experience and senior candidates, plus scenario questions.

This set is written for DevOps work: shipping pipeline, container and cloud data into Splunk and writing searches and alerts on it. For production troubleshooting drills, read real-time Splunk interview questions; for longer incident stories, Splunk scenario-based questions; and for clustering and architecture depth, advanced Splunk questions for architects.

Key takeaways

  • Learn the default ports: 9997 for forwarder-to-indexer traffic, 8089 for the management port (splunkd REST API), 8088 for HEC and 8000 for Splunk Web.
  • Know the data pipeline: input, parsing, indexing, search. Interviewers ask which settings apply at which stage.
  • SPL basics (stats, timechart, eval, rex, where, table) appear in almost every round.
  • For Kubernetes, Splunk now points users to the Splunk Distribution of the OpenTelemetry Collector. Splunk Connect for Kubernetes reached end of support in January 2024.
  • Cisco completed its acquisition of Splunk in March 2024; the products keep the Splunk name.
  • Answers were checked against Splunk documentation in November 2026.
LevelQuestionsFocus
Fresher1 to 28Components, ports, indexes, basic SPL
2 to 5 years29 to 54HEC, props/transforms, alerts, Kubernetes, CI/CD data
Senior55 to 66Scaling, clustering, cost, governance
Scenarios67 to 76Troubleshooting

Questions for freshers

1. What is Splunk?

Splunk is a platform for collecting, indexing, searching and visualising machine data such as logs, metrics and events. It is available as Splunk Enterprise (self-managed) and Splunk Cloud Platform (managed by Splunk).

2. Why do DevOps teams use Splunk?

To search logs from many systems in one place, build dashboards for services and pipelines, alert on errors, and investigate incidents and security events.

3. What are the main Splunk components?

Forwarders send data, indexers parse and store it, search heads run searches and dashboards, and management roles such as the deployment server, cluster manager and license manager coordinate the deployment.

4. What is a universal forwarder?

A lightweight agent that collects data from files, directories, Windows event logs or scripts and forwards it to indexers. It does minimal processing and has no Splunk Web interface.

5. What is a heavy forwarder?

A full Splunk Enterprise instance configured to forward. It can parse, filter and route data before sending it, at the cost of more resources.

6. Which port do forwarders use to send data to indexers?

Port 9997 by convention. You enable receiving on the indexer on that port and point the forwarder's outputs.conf at it.

7. What are ports 8000, 8089 and 8088 used for?

8000 is Splunk Web, 8089 is the splunkd management port and REST API, and 8088 is the default HTTP Event Collector port.

8. What is an index?

A repository where Splunk stores indexed data. Separating data into indexes (for example app_prod, k8s, security) controls access and retention.

9. What is an event?

A single record of data in Splunk, such as one log line or a multi-line stack trace, with a timestamp and default fields.

10. What are the default fields host, source and sourcetype?

host is the machine that produced the data, source is the file path or input, and sourcetype identifies the data format, which controls how Splunk parses it.

11. What is SPL?

Search Processing Language, Splunk's query language. Commands are chained with pipes, much like a Unix shell.

12. Write a search that counts errors per host in the last hour.

index=app_prod log_level=ERROR earliest=-1h
| stats count by host
| sort - count

13. What does the stats command do?

It computes aggregate statistics such as count, avg, sum, dc (distinct count) and percentiles, optionally grouped with by.

14. What is the difference between stats and timechart?

stats returns a table of aggregates. timechart aggregates over time buckets, with _time on the x-axis, ready to chart.

15. What does eval do?

It creates or changes a field using an expression, for example eval duration_s=duration_ms/1000.

16. What does rex do?

It extracts fields at search time using a regular expression with named groups.

index=web | rex field=_raw "status=(?<status>\d{3})" | stats count by status

17. What is the difference between where and search in a pipeline?

search filters with Splunk search syntax and wildcards. where evaluates an expression, such as where status>=500, and compares fields.

18. What does table do?

It returns only the listed fields as a table, for example | table _time host status uri.

19. What is dedup used for?

It removes events with duplicate values in the given fields, keeping the first one found.

20. What is a saved search?

A search stored with a name so it can be rerun, scheduled, used in a dashboard or turned into an alert or report.

21. What is a dashboard in Splunk?

A collection of panels backed by searches. Splunk offers Dashboard Studio and the older Simple XML (Classic) dashboards.

22. What is an alert?

A saved search that runs on a schedule or in real time and triggers actions, such as email, a webhook or an incident tool, when results meet a condition.

23. What is a lookup?

A table, often CSV or KV store, that adds fields to events. For example, map a host to its owning team.

24. What is a field extraction?

The process of pulling named fields from raw events, either automatically (key=value, JSON) or with configured regex or delimiters.

25. What are search-time and index-time operations?

Index-time operations happen when data is stored (timestamping, line breaking, sourcetype assignment). Search-time operations happen when you search (most field extractions, lookups, tags). Prefer search-time where possible because it is easier to change.

26. What is a Splunk app?

A package of configurations, dashboards, searches and sometimes inputs for a purpose. Add-ons (TAs) usually handle data collection and parsing for a specific technology.

27. What is Splunkbase?

Splunk's catalog of apps and add-ons, both Splunk-built and third party.

28. How is Splunk licensed in general?

Splunk offers ingest-based and workload-based pricing models. Check Splunk's site for current terms; interviewers mainly want you to know that ingest volume matters.

Questions for 2 to 5 years of experience

29. What is the HTTP Event Collector (HEC)?

HEC accepts events over HTTP or HTTPS using a token, without a forwarder. It is the usual way for applications, CI/CD tools, cloud services and collectors to send data to Splunk.

30. Send a test event to HEC.

curl https://splunk.example.internal:8088/services/collector/event \
  -H "Authorization: Splunk $HEC_TOKEN" \
  -d '{"event": "deploy finished", "sourcetype": "cicd:deploy", "index": "cicd"}'

31. What is the difference between the HEC event and raw endpoints?

/services/collector/event expects JSON with an event field and optional metadata. /services/collector/raw accepts raw text that Splunk line-breaks like a normal input.

32. What is indexer acknowledgment in HEC?

An option where HEC confirms that events were indexed, so clients can retry on failure. It requires the client to use channel IDs and poll for acknowledgment.

33. What is props.conf used for?

It sets processing rules per sourcetype, source or host: line breaking, timestamp recognition, character set, and search-time extractions.

34. What is transforms.conf used for?

It defines transformations referenced from props.conf: regex field extractions, routing events to an index, masking data, or dropping events to the null queue.

35. How do you drop noisy debug events before indexing?

On a heavy forwarder or indexer, use a transform that sends matching events to nullQueue.

# props.conf
[app:json]
TRANSFORMS-drop_debug = drop_debug

# transforms.conf
[drop_debug]
REGEX = "level":"DEBUG"
DEST_KEY = queue
FORMAT = nullQueue

36. How do you set correct timestamps for a custom log format?

In props.conf, set TIME_PREFIX, TIME_FORMAT and MAX_TIMESTAMP_LOOKAHEAD, and set TZ if the logs lack a time zone.

37. How do you handle multi-line stack traces?

Set SHOULD_LINEMERGE = false and a LINE_BREAKER regex that matches the start of each new event, such as a timestamp. This is faster than line merging.

38. What is the deployment server?

A Splunk instance that distributes apps and configuration to forwarders grouped into server classes, so you manage thousands of forwarders centrally.

39. How do you collect Kubernetes logs into Splunk today?

Deploy the Splunk Distribution of the OpenTelemetry Collector with its Helm chart. It collects container logs, metrics and traces and sends logs to Splunk Platform through HEC. Splunk Connect for Kubernetes reached end of support in January 2024.

40. How do you get AWS data into Splunk?

Options include the Splunk Add-on for AWS (pulling CloudWatch, CloudTrail, S3 and more), Amazon Data Firehose delivering to a HEC endpoint, and Lambda-based forwarding.

41. How do you send CI/CD events to Splunk?

Post build and deploy events to HEC from the pipeline, or use vendor add-ons for tools such as Jenkins and GitHub. Then chart deployment frequency and failure rate per service.

42. What is a sourcetype and why define your own?

A sourcetype tells Splunk how to parse data. Defining your own for custom application logs gives correct line breaking, timestamps and field extractions.

43. What is tstats and why is it fast?

tstats runs statistics on indexed fields and accelerated data models instead of raw events, so it is much faster for large time ranges.

| tstats count where index=app_prod by sourcetype, host

44. What is a data model?

A structured, hierarchical description of data, such as the Common Information Model (CIM) data models for web or authentication. Accelerated data models speed up pivots and tstats.

45. What is CIM?

The Common Information Model, a set of standard field names and data models so that data from different vendors can be searched the same way.

46. What are summary indexing and report acceleration?

Both precompute results. Summary indexing writes scheduled search results into a summary index; report acceleration builds summaries automatically for qualifying reports.

47. What is the difference between real-time and scheduled alerts?

Scheduled alerts run on a cron schedule over a time window. Real-time alerts run continuously and use more resources, so most teams prefer scheduled alerts every few minutes.

48. What is alert throttling?

Suppressing repeat triggers for a period, optionally per field value such as host, so one problem does not send hundreds of notifications.

49. How do you correlate events across services?

Use a shared field such as a trace or request ID and group with stats values(...) by trace_id or transaction. stats is usually faster than transaction.

50. What does the btool command do?

splunk btool props list --debug shows the merged configuration and which file each setting comes from. It is the first tool for configuration precedence problems.

51. What is the KV store?

A key-value store inside Splunk (based on MongoDB) used for lookups and app state that change often.

52. How do you mask sensitive data?

At index time, use SEDCMD in props.conf or a transform to rewrite matching values. Better still, avoid logging secrets in the application.

53. What is the Splunk REST API used for?

On port 8089, it lets scripts run searches, manage saved searches, inputs and users. Automation and CI tools use it to manage Splunk as code.

54. What is Splunk Observability Cloud?

Splunk's separate SaaS product for infrastructure monitoring, APM, RUM and synthetics, built on OpenTelemetry. Splunk Platform (Enterprise or Cloud) is the log and search product.

Questions for senior candidates

55. What is an indexer cluster?

A group of peer indexers managed by a cluster manager that replicate data. The replication factor sets how many copies of raw data exist; the search factor sets how many are searchable.

56. What is a search head cluster?

A group of search heads that share knowledge objects and scheduled search load, coordinated by an elected captain, with a deployer pushing apps.

57. What are hot, warm, cold and frozen buckets?

Buckets age through stages: hot (being written), warm (read-only, recent), cold (older, often cheaper storage) and frozen (deleted or archived). Retention settings in indexes.conf control the movement.

58. What is SmartStore?

An indexer feature that keeps warm data in remote object storage such as S3, with a local cache, separating storage from compute.

59. How do you size a Splunk deployment?

Start from daily ingest volume, retention, search concurrency and data growth, then follow Splunk's reference hardware and validated architectures. Search load often limits more than ingest does.

60. How do you control ingest cost?

Drop or sample noisy data at the edge, route debug logs to cheaper storage or shorter retention, avoid duplicate inputs, and review license usage per sourcetype and index regularly.

61. How would you manage Splunk configuration as code?

Store apps and .conf files in Git, test changes in a non-production environment, and deploy through the deployment server, cluster manager or deployer from CI. Use btool checks in the pipeline.

62. How do you secure a Splunk deployment?

Enable TLS for forwarder and HEC traffic, use SSO with role-based access, restrict indexes per role, rotate HEC tokens, and limit access to the management port.

63. How do you design index strategy for a multi-team company?

Split indexes by access boundary and retention need (for example per team and environment), keep sourcetypes consistent, and avoid too many tiny indexes, which add management overhead.

64. How do you detect a forwarder that silently stopped sending?

Schedule a search on internal metrics, for example | tstats latest(_time) where index=* by host, and alert when a host's last event is older than expected. The Monitoring Console also tracks missing forwarders.

65. When would you choose Splunk over the ELK stack?

When you want a supported commercial platform with mature SPL, security content and enterprise support, and budget allows. ELK or OpenSearch suits teams that want open-source control and can run the clusters.

66. How does Splunk fit with OpenTelemetry?

Splunk's OpenTelemetry Collector distribution collects logs, metrics and traces; logs go to Splunk Platform through HEC and metrics and traces to Splunk Observability Cloud. That lets teams use vendor-neutral instrumentation.

Scenario questions

67. A forwarder shows as connected but no data arrives. What do you check?

Check inputs.conf paths and permissions, whether the file was already read (fishbucket), the target index exists and the role can see it, and splunkd.log on the forwarder for errors.

68. Events show the wrong time. How do you fix it?

Fix the timestamp settings in props.conf for that sourcetype (TIME_FORMAT, TIME_PREFIX, TZ) on the first full Splunk instance that parses the data, then reindex if needed.

69. HEC returns 403. What is wrong?

The token is invalid or disabled, or the header format is wrong. It must be Authorization: Splunk <token>. Also confirm the token allows the target index.

70. HEC returns 503 during a traffic spike. What do you do?

The indexers are back-pressured or the queue is full. Add HEC capacity behind a load balancer, check indexer queue fill in the Monitoring Console, and make clients retry with backoff.

71. A dashboard is slow. How do you speed it up?

Use base searches shared by panels, switch to tstats on accelerated data, narrow time ranges and indexes, and avoid leading wildcards and transaction.

72. Licence usage doubled overnight. How do you investigate?

Search the license usage logs in _internal grouped by sourcetype, index and host to find the source, then check for a debug log level left on or a duplicated input.

73. Multi-line Java exceptions are split into many events. Fix?

Set LINE_BREAKER to split only before lines that start with a timestamp, and SHOULD_LINEMERGE = false, for that sourcetype.

74. After a deploy, error alerts fire for one Kubernetes namespace. How do you narrow it down?

Search by namespace and pod with timechart count by k8s.pod.name, compare versions using the deploy events from CI, and check container restarts in the collector's metrics.

75. Duplicate events appear in an index. Why?

Common causes: two inputs reading the same file, a forwarder sending to two output groups, or log rotation with crcSalt settings causing reread. Find duplicates with stats count by _raw.

76. An alert email never arrives. What do you check?

Whether the search returns results for the window, the trigger condition, throttling, the scheduler logs in _internal for skipped searches, and the mail server settings.

How to prepare

  • Install Splunk Enterprise (free trial) and a universal forwarder on a VM. Index Linux and Nginx logs.
  • Write 20 SPL searches from memory: errors per host, p95 latency, top URIs, failed logins.
  • Send events from a CI job to HEC and build a deployment dashboard.
  • Read the props.conf and transforms.conf spec files; mid-level rounds test them.

Our ELK Stack training in Pune (Kharadi classroom or live online) teaches the same log pipeline concepts with open-source tools, which helps with comparison questions. You can also read 20 log aggregation tools by pipeline layer and the 80+ ELK Stack interview questions.

FAQ

What are the default Splunk ports?

8000 for Splunk Web, 8089 for the management port and REST API, 8088 for HTTP Event Collector, and 9997 by convention for forwarders sending to indexers.

What is the difference between a universal and heavy forwarder?

A universal forwarder is a lightweight agent that only collects and sends. A heavy forwarder is a full Splunk instance that can parse, filter and route data before forwarding.

What SPL commands should I know for interviews?

stats, timechart, eval, rex, where, table, sort, dedup, top, lookup and tstats cover most interview questions.

Is Splunk Connect for Kubernetes still supported?

No. It reached end of support in January 2024. Splunk recommends the Splunk Distribution of the OpenTelemetry Collector for Kubernetes.

Who owns Splunk now?

Cisco completed its acquisition of Splunk in March 2024. Splunk products keep the Splunk name.

What is HEC used for?

The HTTP Event Collector lets applications, scripts and collectors send events to Splunk over HTTP or HTTPS with a token, without installing a forwarder.

Are there Splunk certifications?

Yes. Splunk publishes its certification tracks on its own training site; check there for current exams. Our Splunk certification interview questions post covers related topics.

References

What's Your Reaction?

Like Like 0
Dislike Dislike 0
Love Love 0
Funny Funny 0
Angry Angry 0
Sad Sad 0
Wow Wow 0
Mridul I am a passionate technology enthusiast with a strong focus on DevOps, Cloud Computing, and Cybersecurity. Through my blogs at DevOps Training Institute, I aim to simplify complex concepts and share practical insights for learners and professionals. My goal is to empower readers with knowledge, hands-on tips, and industry best practices to stay ahead in the ever-evolving world of DevOps.