NetApp storage Installation

In case of usage of Virtual Appliance
  • Use local account lpar2rrd for hosting of STOR2RRD on the virtual appliance
  • Use /home/stor2rrd/stor2rrd as the product home
  • Use lpar2rrd account on storages as it is configured in /home/stor2rrd/stor2rrd/etc/stor2rrd.cfg (STORAGE_USER)
  • Replace lpar2rrd by stor2rrd user in all examples below.

The program uses NetApp CLI (ssh) interface for getting of performance data and NetApp API (http[s]) for getting of configuration data from the storage. It does not require installation of any aditional software.
One of bigest advantage of our solution is not requring storage admin rights for getting data from the storage!
  • Allow TCP/IP communication from STOR2RRD host to all NetApp storages on port 22 and 443 (80 instead of 443 if you do not use ssl)

  • Read this in case of NetApp Ontap v9 support.

  • Create ssh-keys on STOR2RRD host under stor2rrd user if do not exist yet.
    Type enter instead of passphrase
    # su - stor2rrd # (use lpar2rrd user on the Appliance)
    $ ssh-keygen -t rsa
      Generating public/private rsa key pair.
      Enter file in which to save the key (/home/stor2rrd/.ssh/id_rsa):
      ...
    $ cat ~/.ssh/id_rsa.pub
    
  • Login to C-mode NetApp system as admin and create user for performance monitoring:
    Cmode::> security login role create -role monitor -cmddirname DEFAULT -access readonly
    Cmode::> security login role create -role monitor -cmddirname set -access all
    Cmode::> security login role create -role monitor -cmddirname statistics -access all
    
    # You can check the role you've just created
    Cmode::> security login role show -role monitor
    
                 Role          Command/                                      Access
      Vserver    Name          Directory                               Query Level
      ---------- ------------- --------- ----------------------------------- --------
      xorux      monitor       DEFAULT                                       readonly
      xorux      monitor       set                                           all
      xorux      monitor       statistics                                    all
      3 entries were displayed.
    
    # Create stor2rrd user, enable SSH, ONTAPI
    Cmode::> security login create -username stor2rrd -role monitor -application ssh -authmethod password
    Cmode::> security login create -username stor2rrd -role monitor -application ontapi -authmethod password
    Cmode::> security login create -username stor2rrd -role monitor -application ssh -authmethod publickey
    
    # Assing stor2rrd user for monitoring role
    Cmode::> security login modify stor2rrd -role monitor *
    
  • Enable ssh key authentication. On the STOR2RRD server show the public key (id_rsa.pub or id_rsa.pub), under stor2rrd (lpar2rrd user on the Appliance)
    $ cat ~/.ssh/id_rsa.pub
    $ ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCu7y41DTAqJwqA4KJfZQWiqU7IMbs8lbK0nvDgZ9dMWmd3l+uawDKxkUdvnRUxBrLqGrxHZx3CjcLoNDqvlFl0AKE3DRTmv6o7plD3b1jRwsCABpNob5knZ/A3E4uVWytJPz036zjkn4vUSrGKJon3oUgLkmtSJoT2FCLvmPbYONPEqs+tQ/yCCkIpVcZwCxwPCbC7nnuuXhjMmRQNzNooREgBHr4eUWr3rlUPNxsSBjg6Mcbfb3u3O9+e49KJKUvMoD40AULW3h3j6bJ6bTTyhKavETpnkryeEq3936oVJdKdd4noraR54rK8lLrErFYgAG8ugLrxzmOYt6dN1phb
    
    Place the key into this command on NetApp (still under admin user)
    Cmode::> security login publickey create -username stor2rrd -publickey "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCu7y41DTAqJwqA4KJfZQWiqU7IMbs8lbK0nvDgZ9dMWmd3l+uawDKxkUdvnRUxBrLqGrxHZx3CjcLoNDqvlFl0AKE3DRTmv6o7plD3b1jRwsCABpNob5knZ/A3E4uVWytJPz036zjkn4vUSrGKJon3oUgLkmtSJoT2FCLvmPbYONPEqs+tQ/yCCkIpVcZwCxwPCbC7nnuuXhjMmRQNzNooREgBHr4eUWr3rlUPNxsSBjg6Mcbfb3u3O9+e49KJKUvMoD40AULW3h3j6bJ6bTTyhKavETpnkryeEq3936oVJdKdd4noraR54rK8lLrErFYgAG8ugLrxzmOYt6dN1phb" -comment STOR2RRD
    

STOR2RRD storage configuration

  • Encrypt password used for storage access (stor2rrd)
    # perl ./bin/spasswd.pl
      Encode password for storage authentication:
      -------------------------------------------
      Enter password:
      Re-enter password:
      Copy the following string to the password field of the corresponding line in etc/storage-list.cfg:
        IT5gYGAK
    
  • Configure storages in etc/storage-list.cfg
    Under stor2rrd account create config entry. Uncomment (remove the hash) example line and adjust it:
    $ vi /home/stor2rrd/stor2rrd/etc/storage-list.cfg
    
    #
    # NetApp
    #
    # storage alias:NETAPP:NetApp mode [CMODE|7MODE]:hostname/IP:ssh port:API port:API uses SSL/HTTPS [0|1]:API user name:API encrypted password:VOLUME_AGG_DATA_LIM:VOLUME_AGG_IO_LIM:SAMPLE_RATE_MINS
    # to encrypt password use: perl ./bin/spasswd.pl
    #cmode-storage-alias:NETAPP:CMODE:192.168.1.1:22:80:0:stor2rrd:IT5gYGAK:1024:10:0
    cmode-storage-alias:NETAPP:CMODE:192.168.1.1:22:80:0:stor2rrd:IT5gYGAK
    
    # https example
    cmode-storage-alias:NETAPP:CMODE:192.168.1.1:22:443:1:stor2rrd:IT5gYGAK
    
    Note: if you want to use https on VMware Virtual Appliance 1.25-1 and less then install this package at first:
    perl-LWP-Protocol-https-6.04-4.el7
    perl-Mozilla-CA-20130114-5.el7
    # yum install perl-LWP-Protocol-https perl-Mozilla-CA-20130114-5
    
  • Assure you have enough of disk space on the filesystem where is STOR2RRD installed
    Roughly you might count 2 - 30 GB per a storage (it depends on number of volumes, 30GB for 5000 volumes)
    $ df -g /home   # AIX
    $ df -h /home   # Linux
    
  • Test the storage connectivity:
    $ cd /home/stor2rrd/stor2rrd
    $ ./bin/config_check.sh 
      =====================================
      STORAGE: cmode-storage-alias : NetApp
      =====================================
      connection ok
    
  • Schedule to run storage agent from stor2rrd crontab (lpar2rrd on Virtual Appliance)
    $ crontab -e
    
    # NetApp FAS : C-mode & 7-mode (same code, use it just once)
    0,5,10,15,20,25,30,35,40,45,50,55 * * * * /home/stor2rrd/stor2rrd/load_netappperf.sh > /home/stor2rrd/stor2rrd/load_netappperf.out
    
    Assure there is already an entry with the GUI creation running once an hour in crontab
    $ crontab -e
    
    # STOR2RRD GUI
    0 * * * * /home/stor2rrd/stor2rrd/load.sh > /home/stor2rrd/stor2rrd/load.out 2>&1
    
  • Let run the storage agent for 15 - 20 minutes to get data, then:
    $ cd /home/stor2rrd/stor2rrd
    $ ./load.sh
    
  • Go to the web GUI: http://<your web server>/stor2rrd/
    Use Ctrl-F5 to refresh the web browser cache.

  • Read this in case of NetApp Ontap v9 support.

Supported devices

Support of E-series with SANtricity API used below works since STOR2RRD v1.35-1

SMcli installation
  • Download and install SMcli package from NetApp download site.
    Go to support.netapp.com ➡ downloads ➡ software ➡ (login) ➡
          E-Series SANtricity Storage Manager (all platforms - Go) ➡
            View & download ➡ Continue (on end of the page) ➡ Accept (on end of the page) ➡
              Download SMIA-<OS platform>-11.10.nnnn.nnnn.bin
  • AIX prerequisites:
    Install X11 packages from your AIX install media:
    • X11.adt
    • X11.base
    • X11.motif
    # lslpp -L |egrep "X11.adt|X11.base|X11.motif"
      X11.adt.bitmaps            6.1.0.0    C     F    AIXwindows Application
      X11.adt.ext                6.1.6.0    C     F    AIXwindows Application
      X11.adt.imake              6.1.6.0    C     F    AIXwindows Application
      X11.adt.include            6.1.4.0    C     F    AIXwindows Application
      X11.adt.lib                6.1.2.0    C     F    AIXwindows Application
      X11.adt.motif              6.1.6.0    C     F    AIXwindows Application
      X11.base.common            6.1.0.0    C     F    AIXwindows Runtime Common
      X11.base.lib               6.1.6.0    C     F    AIXwindows Runtime Libraries 
      X11.base.rte               6.1.6.0    C     F    AIXwindows Runtime Environment
      X11.base.smt               6.1.6.0    C     F    AIXwindows Runtime Shared
      X11.base.xpconfig          6.1.0.0    C     F    Xprint Configuration Files 
      X11.motif.lib              6.1.6.0    C     F    AIXwindows Motif Libraries 
      X11.motif.mwm              6.1.6.0    C     F    AIXwindows Motif Window
    
  • If you are on Virtual Appliance < 1.25-1 then run this cmd to clean it out otherwise installation fails
    # rpm -e SMruntime-10.86.0A05.0001-1.x86_64 SMclient-10.86.0G05.0043-1.noarch SMesm-10.86.0G05.0007-1.noarch
    
  • If there is already installed SMcli from IBM to monitor DS3000/4000/5000 storages then it must be removed:
    # /opt/IBM_DS/Uninstall\ IBM\ System\ Storage\ DS\ Storage\ Manager\ 10/Uninstall\ IBM\ System\ Storage\ DS\ Storage\ Manager\ 10
    
  • Install SMcli under root user (Installation manual):
    # umask 022
    # chmod 775 SMIA-LINUXX64-11.30.0A00.0010.bin
    # ./SMIA-LINUXX64-11.30.0A00.0010.bin  -i console
    
      Please choose the Install Set to be installed by this installer.
        1- Typical (Full Installation)
    --> 2- Management Station
        3- Host
        4- Customize...
    
      ENTER THE NUMBER FOR THE INSTALL SET, OR PRESS <ENTER> TO ACCEPT THE DEFAULT
      : 2
      ......
    
      Installation Complete
      ---------------------
      Congratulations. SANtricity has been successfully installed to:
    
      /opt/StorageManager
    
    # ls -lL /usr/bin/SMcli
      -rwxr-xr-x 1 root root 1108 Oct 14 21:44 /usr/bin/SMcli
    
    In case of any problem check Install logs in /opt/StorageManager/InstallLogs/ or /opt/InstallLogs

  • Stop SMmonitor service
    AIX:
    # SMmonitor stop
    # rmitab smmonitor
    
    Linux:
    # /opt/SMgr/client/SMmonitor stop 
    # chkconfig --list SMmonitor 
    # chkconfig --level 2345 SMmonitor off 
    alternatively: 
      # rm -f /etc/rc2.d/S99SMmonitor /etc/rc.d/init.d/SMmonitor
    
    Linux systemd:
    # systemctl stop smmonitor.service
    # systemctl disable smmonitor.service
    
  • Allow user stor2rrd run SMcli API (lpar2rrd user on Virtual Appliance)
    Under root execute following:
    # rm -f /var/opt/SM/emwdata_v04_bin.lock /var/opt/SM/pref_01_bin.lock
    # chown stor2rrd /opt/SMgr/client/SMcli /var/opt/SM/prefbackup_01.bin 
    # chown stor2rrd /var/opt/SM/emwback_v04.bin /var/opt/SM /var/opt/SM/emwdata_v04.bin
    # chown stor2rrd /var/opt/SM /var/opt/SM/pref_01.bin
    # chmod 755 /opt/SMgr/client/lib/*
    
    If you do not do it then you get such error by running it under someone else than root:
    (Do not forget to deactivate SMmonitor as above as it can change file rights back to root itself)
    # su - stor2rrd
    $ SMcli -d
      An internal error, error code 12, has occurred.  This is possibly due to
      initialization problems when loading the necessary internal files.  Please check
      your installation of the storage management client. If problems continue 
      to persist, re-install the storage management client.
    

Storage configuration

  • Open TCP port 2463 to the storages if there is firewall between (your host ➞ storage:2463 )

  • Connect new storage
    # su - stor2rrd
    $ SMcli -d
      There are currently no storage subsystems listed in the configuration file. Add
      storage subsystems using the Add Storage Subsystem option in the storage management
      software or by command line.
    $ SMcli -A 1.1.1.2 1.1.1.3
      New storage subsystem was discovered at address 1.1.1.2
      New storage subsystem was discovered at address 1.1.1.3
      SMcli completed successfully
    $ SMcli -d
      E5500-01  ControlerA_hostname  ControlerB_hostname
    
    Use "E5500-01" further for etc/storage-list.cfg configuration

  • Check if the storage needs user authorization (new firmwares allows to specify password for monitor user role):
    $ SMcli -n E5500-01 -e -c "show storageSubsystem summary;"
    
    When it fails then use "monitor" role credetials (password can be set in the storage GUI):
    $ SMcli -n E5500-01 -e -c "show storageSubsystem summary;"
      The operation needs either the 'Administrator' or 'Monitor' password to be set.
    $ SMcli -n E5500-01 -p <password> -R monitor -e -c "show storageSubsystem summary;"
    
    Use that credentials in future etc/storage-list.cfg configuration

STOR2RRD storage configuration

  • Configure storages in etc/storage-list.cfg
    Under stor2rrd account create config entry. Uncomment (remove the hash) example line and adjust it:
    $ vi /home/stor2rrd/stor2rrd/etc/storage-list.cfg  
    
    #
    # NetApp E-series (use DS5K as storage type)
    #
    #storage_alias:DS5K:storage user:user_password:VOLUME_AGG_DATA_LIM:VOLUME_AGG_IO_LIM:SAMPLE_RATE_MINS
    #new storages can be configured to use username/password (use "monitoring" account), old ones do not have this option
    #
    #DS3700:DS5K:monitor:password:256:10:5
    #DS5020:DS5K:
    E5500-01:DS5K:
    
  • Assure you have enough of disk space on the filesystem where is STOR2RRD installed
    Roughly you might count 2 - 30 GB per a storage (it depends on number of volumes, 30GB for 5000 volumes)
    $ df -g /home   # AIX
    $ df -h /home   # Linux
    
  • Test the storage connectivity:
    $ cd /home/stor2rrd/stor2rrd
    $ ./bin/config_check.sh 
      =======================
      STORAGE: E5500-01: DS5K
      =======================
      /usr/SMclient/SMcli -n E5500-01 -e -c "storageSubsystem summary;"
      connection ok
    
  • Schedule to run storage agent from stor2rrd crontab (lpar2rrd on Virtual Appliance)
    $ crontab -e  
    
    # NetApp E-series (use same script load_ds5perf.sh as IBM DS3000/DS4000/DS5000)
    0,5,10,15,20,25,30,35,40,45,50,55 * * * * /home/stor2rrd/stor2rrd/load_ds5perf.sh > /home/stor2rrd/stor2rrd/load_ds5perf.out
    
    Assure there is already an entry with the GUI creation running once an hour in crontab
    $ crontab -e
    
    # STOR2RRD GUI
    0 * * * * /home/stor2rrd/stor2rrd/load.sh > /home/stor2rrd/stor2rrd/load.out 2>&1
    
  • Let run the storage agent for 15 - 20 minutes to get data, then:
    $ cd /home/stor2rrd/stor2rrd
    $ ./load.sh
    
  • Go to the web GUI: http://<your web server>/stor2rrd/
    Use Ctrl-F5 to refresh the web browser cache.
The program uses NetApp CLI (ssh) interface for getting of performance data and NetApp API (http[s]) for getting of configuration data from the storage. It does not require installation of any aditional software.
One of bigest advantage of our solution is not requring storage admin rights for getting data from the storage!

  • Allow TCP/IP communication from STOR2RRD host to all NetApp storages on port 22 and 443 (80 instead of 443 if you do not use ssl)

  • Create ssh-keys on STOR2RRD host under stor2rrd user if do not exist yet.
    Type enter instead of passphrase
    # su - stor2rrd # (use lpar2rrd user on the Appliance)
    $ ssh-keygen -t rsa
      Generating public/private rsa key pair.
      Enter file in which to save the key (/home/stor2rrd/.ssh/id_rsa):
      ...
    $ cat ~/.ssh/id_rsa.pub
    
  • Login to 7-mode NetApp system as admin and prepare the role for performance monitoring:
    # perf monitoring role - login via ssh / http API, stats, some API commands (aggr-list-info, volume-list-info...)
    7mode> useradmin role add monitor -a cli-stats,cli-man,cli-?,cli-rdfile,cli-sysconfig,login-ssh,login-http-admin,api-aggr-list-info,api-volume-list-info,api-disk-list-info,api-lun-list-info,api-diagnosis-status-get,api-perf-*
    
    # group with role
    7mode> useradmin group add Monitor -r monitor
    
    # user in group
    7mode> useradmin user add stor2rrd -g Monitor
    
    # check if SSH is enabled, public keys can be used and your guest has ssh access:
    # If these values are set in a different way, a secureadmin setup should be done
    # If ssh.access contains list of host then add your STOR2RRD host there
    7mode> options ssh
      ssh.access                   *
      ssh.enable                   on
      ssh.pubkey_auth.enable       on
      ssh2.enable                  on
    
    # check trusted hosts list, there should be * by default or add to the list STOR2RRD host
    7mode> options trusted
      options trusted.hosts        *
    
    Old 7mode firmwares do not support api-diagnosis-status-get in "useradmin role add monitor" cmd and fails with:
    Invalid capabilities: api-diagnosis-status-get
    Could not modify role <monitor>. Error: Invalid capability
    
    Use this cmd instead (STOR2RRD storage status in healthcheck will not work then)
    7mode> useradmin role add monitor -a cli-stats,cli-man,cli-?,cli-rdfile,cli-sysconfig,login-ssh,login-http-admin,api-aggr-list-info,api-volume-list-info,api-disk-list-info,api-lun-list-info,api-perf-*
    
  • Enable ssh key authentication.
    At first check what is your "trusted host", they are listed in vol0 line after string "root=". example below shows that it is 192.168.1.53
    7mode> exportfs
      /vol/vol0 -sec=sys,rw,root=192.168.1.53,nosuid
      ...
    
    You have to log on as root to one of storage configured trusted hosts, mount storage's root filesystem and append the ssh key there.
    NetApp description
    # mount netapp-7mode:/ /mnt
    # cat /home/stor2rrd/.ssh/id_rsa.pub >> /mnt/etc/sshd/stor2rrd/.ssh/authorized_keys2
    # umount /mnt
    
    Note for XoruX Virtual Appliance 1.25-1 and less: there is no NFS mount support therefore above cmd fails.
    If you want to use it there (preferably use already existing trusted host) then install NFS support at first:
    # yum install nfs-utils
    

STOR2RRD storage configuration

  • Encrypt password used for storage access (stor2rrd)
    # perl ./bin/spasswd.pl
      Encode password for storage authentication:
      -------------------------------------------
      Enter password:
      Re-enter password:
      Copy the following string to the password field of the corresponding line in etc/storage-list.cfg:
        IT5gYGAK
    
  • Configure storages in etc/storage-list.cfg
    Under stor2rrd account create config entry. Uncomment (remove the hash) example line and adjust it:
    $ vi /home/stor2rrd/stor2rrd/etc/storage-list.cfg
    
    #
    # NetApp
    #
    # storage alias:NETAPP:NetApp mode [CMODE|7MODE]:hostname/IP:ssh port:API port:API uses SSL/HTTPS [0|1]:API user name:API encrypted password:VOLUME_AGG_DATA_LIM:VOLUME_AGG_IO_LIM:SAMPLE_RATE_MINS
    # to encrypt password use: perl ./bin/spasswd.pl
    #7mode-storage-alias:NETAPP:7MODE:192.168.1.2:22:80:0:stor2rrd:IT5gYGAK:1024:10:0
    7mode-storage-alias:NETAPP:7MODE:192.168.1.2:22:80:0:stor2rrd:IT5gYGAK
    
    # https example
    7mode-storage-alias:NETAPP:7MODE:192.168.1.2:22:443:1:stor2rrd:IT5gYGAK
    
    Note: if you want to use https on VMware Virtual Appliance 1.25-1 and less then install this package at first:
    perl-LWP-Protocol-https-6.04-4.el7
    perl-Mozilla-CA-20130114-5.el7
    # yum install perl-LWP-Protocol-https perl-Mozilla-CA-20130114-5
    
  • Assure you have enough of disk space on the filesystem where is STOR2RRD installed
    Roughly you might count 2 - 30 GB per a storage (it depends on number of volumes, 30GB for 5000 volumes)
    $ df -g /home   # AIX
    $ df -h /home   # Linux
    
  • Test the storage connectivity:
    $ cd /home/stor2rrd/stor2rrd
    $ ./bin/config_check.sh 
      =====================================
      STORAGE: 7mode-storage-alias : NetApp
      =====================================
      connection ok
    
  • Schedule to run storage agent from stor2rrd crontab (lpar2rrd on Virtual Appliance)
    $ crontab -e
    
    # NetApp FAS : C-mode & 7-mode (same code, use it just once)
    0,5,10,15,20,25,30,35,40,45,50,55 * * * * /home/stor2rrd/stor2rrd/load_netappperf.sh > /home/stor2rrd/stor2rrd/load_netappperf.out
    
    Assure there is already an entry with the GUI creation running once an hour in crontab
    $ crontab -e
    
    # STOR2RRD GUI
    0 * * * * /home/stor2rrd/stor2rrd/load.sh > /home/stor2rrd/stor2rrd/load.out 2>&1
    
  • Let run the storage agent for 15 - 20 minutes to get data, then:
    $ cd /home/stor2rrd/stor2rrd
    $ ./load.sh
    
  • Go to the web GUI: http://<your web server>/stor2rrd/
    Use Ctrl-F5 to refresh the web browser cache.