How to set up Apache Storm on mac using brew

I am learning about Apache Storm and the guide had long and winding instructions on how to install Apache Storm.  I installed the single node using brew and it was a breeze. I did not had to spend time figuring out out if I was installing latest version.

Pre-requisites:

Make sure that you have ‘HomeBrew‘ installed. One reason that I prefer ‘brew’ is that I do not have to remember where things are installed and brew manages and fixes any updates that have been introduced. It also makes installation and uninstall super easy. Everything this is saved under

/url/local/Cellar.

Step 1: Install Zookeeper

‘ brew install zookeeper’

After the installation is complete, you will see this message.

To have launchd start zookeeper at login:
ln -sfv /usr/local/opt/zookeeper/*.plist ~/Library/LaunchAgents
Then to load zookeeper now:
launchctl load ~/Library/LaunchAgents/homebrew.mxcl.zookeeper.plist
Or, if you don’t want/need launchctl, you can just run:
zkServer start

I prefer not to launch zookeeper at login as it will sow down my mac login. So I prefer to start and stop Zookeeper using the command-
zkServer start/stop’

Step 2: Install ZeroMQ

‘brew install zeromq’

Step 3: Install Apache storm

‘brew install storm’

This above command will install everything in Cellar folder and will have a symlink in ‘/usr/local/opt/storm’ folder.

Step 4: Update the storm config file storm.yaml.

This file will be in /usr/local/opt/storm/libexec/conf folder.
After opening this file you will see that everything is commented out. So we will add the following.

storm.zookeeper.servers:
– “localhost”
# – “server2”
#
nimbus.host: “localhost”
nimbus.thrift.port: 6627
ui.port: 8772
storm.local.dir: “/Users/dinesharora/storm/data”
java.library.path: /usr/lib/jvm”
supervisor.slots.ports:
– 6700
– 6701
– 6702
– 6703

If you will notice that I have added “storm.local.dir”. Make sure that whatever folder you create, they have the right permissions because I initially added “/home/user/storm/data” and did not realize that you cannot create any directory under “/home” folder. This was causing issues when I started nimbus and supervisor.

Step 5: start zookeeper, nimbus, supervisor and ui

Below are the location on shell scripts. Start them in the given sequence

‘zkServer start’
‘/usr/local/opt/storm/libexec/bin/storm nimbus’
‘/usr/local/opt/storm/libexec/bin/storm supervisor’
‘/usr/local/opt/storm/libexec/bin/storm ui’

Step 6: Check that everything is running smoothly.

To do that run the below command.
‘jps’

The above command should give the following result.

Dineshs-MBP:bin dinesharora$ jps
5282 supervisor
5267 nimbus
5460 core
5735 Jps
4235 QuorumPeerMain

Step 7: Check the server index page. Access the url – ‘http://localhost:8772/index.html’.

If you are able to access the page then your installation is successful.

 

P.S. If you upgraded to new OSX – El Captain then you might have issues with brew itself. Do yourself a favor and reinstall brew again.

 

~ Stir a storm

  • Jose Garcia

    is it /usr/local/cellar instead? I tried this installation on Sierra. When I run nimbus it just stays there. I suspect the problem could be on java.library.path. I used the /Library/Java/JavaVirtualmachines. If I try to use the one you used it does not exist. Any advice? thank you. Jose

    • It seems like sym link is not created for you so Cellar folder work as well. Use mdfind command to confirm.

      • Jose Garcia

        mdfind returns /usr/local/Cellar. I set up Javahome to the library path. When I run nimbus I have to stop it and run it again. It takes ages to start running. Then I start the supervisor, I get a list of scripts and java files and stays there. I don;t get the chance to continue the start sequence

  • Ross Mills

    This site has been very helpful. At first, it worked great. However, from time to time, my storm cluster would have connection issues with very cryptic error messages. After many days of struggling with these, I finally found a way to prevent the errors from happening. I’d like to recommend that you include the following 2 lines in your storm.yaml file. It resolved the issue for me:

    storm.local.hostname: “localhost”
    nimbus.seeds : [“localhost”]