Skip to content

Commit 2e2e9c5

Browse files
nipuntalukdarnipuntalukdar
committed
Added module for collecting stats from Storm topologies
1 parent 9f740e8 commit 2e2e9c5

3 files changed

Lines changed: 547 additions & 0 deletions

File tree

‎storm_topology/README.mkdn‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
Storm Topology
2+
===============
3+
4+
Python module for getting stats for Storm topologies
5+
6+
This module gets stats for Storm Topologies from Nimbus host.
7+
The communication to Storm Nimbus uses Thrift protocol.
8+
We have to install python thrift. Execute below commnand if thrift is not already installed:
9+
$sudo pip install thrift
10+
11+
Please install thrif-compiler if you want to generate the thrift modules. In that case,
12+
download Apache Storm from [github](https://github.com/apache/storm) and then run the below command from storm/storm-core/src directory:
13+
thrift -gen py storm.thrift
14+
The generated modules will be under gen-py folder. Copy the content of the folder to /usr/lib/ganglia/stormpy.
15+
The stormpy directory may be under some directory other than /usr/lib/ganglia. For example, we may copy the content to /home/someusername/stormpy. Then we specify the same in storm_topology.pyconf in the parameter storm_thrift_gen. In this case the value of storm_thrift_gen will be "/home/someusername".
16+
17+
Storm source already contains generated thrift code for Python. It is accessible [here](https://github.com/apache/storm/tree/master/storm-core/src/py). So, we may directly download the content from [here](https://github.com/apache/storm/tree/master/storm-core/src/py) if we want an easy way out.
18+
19+
20+
Topology names serve as the group names of the sats published.
21+
This module collects stats for uptime, worker count, task count, process latency, execute latency etc.
22+
For example if there is a topology SomeTopology which runs a spout "Spout" and bolts bolta,bolb,boltd then
23+
below stats are published:
24+
25+
* SomeTopology_Spout_Tasks
26+
* SomeTopology_Spout_Executors
27+
* SomeTopology_Spout_Emitted
28+
* SomeTopology_Spout_Acked
29+
* SomeTopology_Spout_Transferred
30+
* SomeTopology_Spout_Failed
31+
* SomeTopology_Spout_CompleteLatency
32+
* SomeTopology_bolta_Failed
33+
* SomeTopology_bolta_Executed
34+
* SomeTopology_bolta_Tasks
35+
* SomeTopology_bolta_ProcessLatency
36+
* SomeTopology_bolta_Executors
37+
* SomeTopology_bolta_Emitted
38+
* SomeTopology_bolta_ExecuteLatency
39+
* SomeTopology_bolta_Transferred
40+
* SomeTopology_bolta_Acked
41+
* SomeTopology_boltb_Failed
42+
* SomeTopology_boltb_Executed
43+
* SomeTopology_boltb_Tasks
44+
* SomeTopology_boltb_ProcessLatency
45+
* SomeTopology_boltb_Executors
46+
* SomeTopology_boltb_Emitted
47+
* SomeTopology_boltb_ExecuteLatency
48+
* SomeTopology_boltb_Transferred
49+
* SomeTopology_boltb_Acked
50+
* SomeTopology_boltd_Failed
51+
* SomeTopology_boltd_Executed
52+
* SomeTopology_boltd_Tasks
53+
* SomeTopology_boltd_ProcessLatency
54+
* SomeTopology_boltd_Executors
55+
* SomeTopology_boltd_Emitted
56+
* SomeTopology_boltd_ExecuteLatency
57+
* SomeTopology_boltd_Transferred
58+
* SomeTopology_boltd_Acked
59+
* SomeTopology_UptimeSecs
60+
* SomeTopology_ExecutorCount
61+
* SomeTopology_WorkerCount
62+
* SomeTopology_TaskCount
63+
64+
## AUTHOR
65+
66+
Author: Nipun Talukdar

‎storm_topology/conf.d/storm_topology.pyconf‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
modules {
2+
module {
3+
name = "storm_topology"
4+
language = "python"
5+
param topologies {
6+
value = "SampleTopology,AnotherTopology"
7+
}
8+
param SampleTopology_spouts {
9+
value = "SampleSpoutTwo"
10+
}
11+
param SampleTopology_bolts {
12+
value = "boltc"
13+
}
14+
param AnotherTopology_spouts {
15+
value = "Spout"
16+
}
17+
param AnotherTopology_bolts {
18+
value = "bolta,boltb,boltd"
19+
}
20+
param storm_thrift_gen {
21+
value = "/usr/lib/ganglia"
22+
}
23+
param loglevel {
24+
value = "INFO"
25+
}
26+
}
27+
}
28+
29+
collection_group {
30+
collect_every = 20
31+
time_threshold = 90
32+
metric {
33+
name_match = ".*"
34+
}
35+
}

0 commit comments

Comments
 (0)