[Solved] Installing OpenHAB/MQTT on Linux Laptop

[papa]

            --------------------------------------------------
Added 1/29/2016:  Like me, do you have an older (originally Windows) laptop or desktop on which you installed Linux & now you want to use it for OpenHAB with Mosquitto?
     The thread below narrates my cry for help after being stuck a while & all we tried to fix it until I saw a clue that led to a solution.

I just finished a new thread where I aimed to document just what worked for my install.

    For that documentation, see my Linux Laptop: How to Install OpenHAB-Mosquitto thread
            -------------------------------------------------------------------------

     DIY Home Automation colleagues,

I need some help getting everything communicating OpenHAB / MQTT wise using my Linux (Ubuntu 14.04) laptop where I'd like to move with openHAB.

Please be very simple & clear.  I'm not very experienced with Linux. :-)

I've installed openHAB using this github.com/openhab/openhab/wiki/Linux---OS-X & copying simple config files to the proper folders /etc/openhab & /usr/share/openhab
     In my browser I'm able to load the user interface using http://localhost:8080/openhab.app?sitemap=My

On my Linux laptop, I've also done these mosquitto installs
    sudo apt-get install mosquitto
    sudo apt-get install mosquitto-clients

Some install instructs mentioned installing something related to python.  Do I need that?  Specifically what & from where?

My Linux laptop has a reserved ip of 192.168.0.206
      So ... in openhab.cfg, is security:netmask=192.168.0.0/24 correct?

The Gateway 2.2 node sketch has the following lines

        byte mqtt_server[] = { 192, 168, 0, 206}; // MQTT broker address
        byte ip[] = { 192, 168, 0 , 212 }; // Gateway address (if DHCP fails)
...

// Wireless settings
#define NODEID 1 // unique node ID in the closed radio network; gateway is 1
#define RFM_SS 8 // Slave Select RFM69 is connected to pin 8
#define NETWORKID 100 // closed radio network ID

In one terminal window, I can enter mosquitto_sub -v -t 'test/topic'
   Then on another Terminal window, when I enter mosquitto_pub -t 'test/topic' -m 'helloWorld'
           test/topic helloWorld displays on the first window

So that seems to indicate that mosquitto is running

My Gateway is connected to my windows computer.  My router DHCP table shows the Gateway's ethernet shield is connected to my network.
    When I run the Arduino IDE's serial monitor, I just see
      Gateway Software Version GW V2.2

Listening at 915 Mhz...
connecting...
connection failed... [many times]    AND the MQTT LED on the Gateway does not light.

So it seems like OpenHAB & Mosquitto are running & the Gateway is trying to connect but they are not linking up.

What ideas do you have about what I've been missing?

Thanks in advance for your attention & help.

[greginkansas]

is this right?
My Gateway is connected to my windows computer. ?

[papa]

Yes, greginkansas, it was plugged into a USB port on my windows computer so I could see the serial monitor output.

I also tried it in the USB port of the Linux computer hosting OpenHAB & mqtt. The mqtt LED still did not light.

The Gateway worked fine connecting to mqtt on another windows computer in another location that I'm away from for several months.

In case debug mode left too little spare memory on the Arduino, I also just now compiled & uploaded the Gateway sketch with debug commented out.

       Still the mqtt LED does not light.

I just opened port 1883 with sudo iptables -A INPUT -p tcp -m tcp --dport 1883 -j ACCEPT
      Still the mqtt LED does not light.

     When I execute the command    sudo lsof -i TCP:1883        ... I get:

COMMAND PID     USER      FD TYPE DEVICE SIZE/OFF NODE      NAME
mosquitto 23383 mosquitto 3u IPv4 7210959 0t0         TCP    *:1883 (LISTEN)
mosquitto 23383 mosquitto 4u IPv6 7210960 0t0         TCP    *:1883 (LISTEN)
mosquitto 23383 mosquitto 5u IPv4 7731385 0t0         TCP    localhost:1883->localhost:54979 (ESTABLISHED)
java        28306 openhab 147u IPv6 7731384 0t0        TCP    localhost:54979->localhost:1883 (ESTABLISHED)

[computourist]

Hi papa,

- check connectivity to your mosquitto server: ping 192.168.0.206
- to see if Mosquitto is running: type sudo ps ax
- Check your mosquitto.conf file (anonymous access allowed ?)

- To check MQTT messages I use my Android phone with MyMQTT app.

[papa]

Thanks, CT, I'll try that later when I get a chance.

[papa]

OK, CT, from my windows laptop on the same network, I pinged the Linux machine hosting mosquitto
It got 4 solid replies

From sudo ps ax on linux machine, I got a long pid list, but particularly
23383 ? Ss 0:03 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

In the mosquitto.conf file, I have #allow_anonymous true (commented out)
but just above that it says # Defaults to true. so seems like anonymous access is allowed, right?

Now what?

[greginkansas]

Ping gateway?
check these
#include <PubSubClient.h>
EthernetClient ethClient;
PubSubClient mqttClient(mqtt_server, 1883, mqtt_subs, ethClient );

fixed ip here

// if (Ethernet.begin(mac) == 0) { // start the Ethernet connection
#ifdef DEBUG
Serial.println("Failed to configure Ethernet using DHCP");
#endif
Ethernet.begin(mac, ip);
//}

Good luck

[papa]

Yes, greginkansas, I checked the following lines & mine read the same
    #include <PubSubClient.h>
    EthernetClient ethClient;
    PubSubClient mqttClient(mqtt_server, 1883, mqtt_subs, ethClient );

            My Gateway sketch says
byte ip[] = { 192, 168, 0 , 212 }; // Gateway address (if DHCP fails)


I changed the following lines to read as you altered them to fix the variable ip as the address ...

// if (Ethernet.begin(mac) == 0) { // start the Ethernet connection
#ifdef DEBUG
Serial.println("Failed to configure Ethernet using DHCP");
#endif
Ethernet.begin(mac, ip);
//}

Then I pinged the Gateway at 192.168.0.212 & I got good response from the pings

But still no mqtt connection between the Gateway & the mqtt server (serial monitor & mqtt LED does not light)

Both gateway & mqtt seem running & connected to the network but not connecting to each other.

Anyone have other ideas what to check?

[papa]

Who can help me use logging for mosquitto so maybe that can give me info on the problem?

I edited & resaved the mosquitto.conf file like so
    log_dest stdout
    log_dest stderr
    log_dest syslog
    log_dest topic
but I don't know the "console" where some are supposed to be output. Where is that?

I also uncommented other lines
     connection_messages true
     log_type error

         Then I used sudo service mosquitto stop
then sudo service mosquitto start [I hoped to restart mosquitto to read the edited .conf file
     Is this right or heading in the right direction?

I looked in the /var/log folder but did not see where those loggings might go. Any help here?

     The Arduino IDE did give me an warning about these lines
char *subTopic = "home/rfm_gw/sb/#"; // MQTT subscription topic ; direction is southbound.
char *clientName = "RFM_gateway"; // MQTT system name of gateway.

    I changed to these lines & it compiled without warning, but mqtt still did not connect.
const char *subTopic = "home/rfm_gw/sb/#"; // MQTT subscription topic ; direction is southbound.
const char *clientName = "RFM_gateway"; // MQTT system name of gateway.

[kmac]

Did you look at the bottom of /var/log/syslog for Mosquitto messages?

The console you are asking about is most likely the terminal screen you are logged into.
On my Ubuntu systems stdout and stderr are set to display on the screen. But I am running Ubuntu server with no GUI, just a command line so yours may be setup different.

[computourist]

The command:

sudo netstat -anp | grep 1883

should show whether Mosquitto is listening on port 1883, and whether you have established MQTT connections.

Again: try to connect with another MQTT client (like Android MyMQTT) to identify the location of the problem (either Mosquitto or RFM_Gateway)

[papa]

Yes, kmac, I'm using a graphical interface & do not see any error messages on terminal.

I powered up the gateway about 6:15am & this is all I see at the bottom of syslog

Jan 25 05:17:01 brent-linux CRON[10683]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jan 25 06:17:01 brent-linux CRON[11148]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jan 25 06:25:01 brent-linux CRON[11221]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Jan 25 07:17:01 brent-linux CRON[11652]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jan 25 07:30:01 brent-linux CRON[11777]: (root) CMD (start -q anacron || :)
Jan 25 07:30:01 brent-linux anacron[11780]: Anacron 2.3 started on 2016-01-25
Jan 25 07:30:01 brent-linux anacron[11780]: Will run job `cron.daily' in 5 min.
Jan 25 07:30:01 brent-linux anacron[11780]: Jobs will be executed sequentially

[papa]

Computourist, this is what I see.  Linux machine with mosquitto is at 192.168.0.206.  Gateway is now at 192.168.0.204. 

$ sudo netstat -anp | grep 1883

tcp        0      0 0.0.0.0:1883            0.0.0.0:*               LISTEN      7068/mosquitto  
tcp        0      0 192.168.0.206:1883      192.168.0.204:49188     FIN_WAIT2   -              
tcp        0      0 192.168.0.206:1883      192.168.0.204:49187     TIME_WAIT   -              
tcp        0      0 192.168.0.206:1883      192.168.0.204:49184     TIME_WAIT   -              
tcp        0      0 192.168.0.206:1883      192.168.0.204:49186     TIME_WAIT   -              
tcp        0      0 127.0.0.1:1883          127.0.0.1:56996         ESTABLISHED 7068/mosquitto  
tcp        0      0 192.168.0.206:1883      192.168.0.204:49185     TIME_WAIT   -              
tcp6       0      0 :::1883                 :::*                    LISTEN      7068/mosquitto  
tcp6       0      0 127.0.0.1:56996         127.0.0.1:1883          ESTABLISHED 28306/java
To me, it looks like mosquitto is listening on port 1883, right?

I'll work on your other suggestion.

[papa]

netstat -tuplen produces:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
...
tcp6 0 0 :::1883 :::* LISTEN 116 8268020

[papa]

        netstat  just generated
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 192.168.0.206:1883 192.168.0.204:49411 FIN_WAIT2
tcp 0 0 192.168.0.206:47545 yw-in-f113.1e100.:https ESTABLISHED
tcp 0 0 192.168.0.206:1883 192.168.0.204:49409 TIME_WAIT
tcp 0 0 192.168.0.206:1883 192.168.0.204:49410 TIME_WAIT
tcp 0 0 192.168.0.206:1883 192.168.0.204:49408 TIME_WAIT
tcp 0 0 localhost:1883 localhost:56996 ESTABLISHED
tcp 0 0 192.168.0.206:52454 yv-in-f189.1e100.:https ESTABLISHED
tcp6 1 0 ip6-localhost:55572 ip6-localhost:ipp CLOSE_WAIT
tcp6 0 0 localhost:56996 localhost:1883 ESTABLISHED
tcp6 0 0 192.168.0.206:44596 li464-188.members:https ESTABLISHED

which shows me the first signs of some possible communication between my linux machine (192.168.0.206) & the gateway (192.168.0.204).

Now if I can figure out what the above results mean, perhaps with someone's help. :-)

[papa]

OK, Computourist, I installed TT3 on my windows machine on my network & am trying to learn using it.

On my Linux machine Terminal I executed: mosquitto_sub -v -t '#' to subscribe to all messages

on TT3, I successfully got a connection with 192.168.0.206 (Linux machine with OpenHAB & mosquitto) port 1883 client TT3
On TT3, I set a topic /hello & message Hello
Then I set a performance test of 100 publications, delay 100 ms

All I got on linux Terminal was a growing stack of ...
home/rfm_gw/sb/node02/dev04 READ
home/rfm_gw/sb/node02/dev02 READ

which are from OpenHAB (specifically a rule), right, not what I published from TT3?

Seems like mosquitto is listening, & messages from inside the linux machine are getting thru,
    but messages from gateway & TT3 are not getting thru

Or am I not using TT3 correctly?

[papa]

By the way the gateway has been powered & connected to my network this whole time

     Executed another netstat & got
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp      0    0     localhost:58199     localhost:1883     ESTABLISHED
tcp 0 0             192.168.0.206:1883         192.168.0.204:49658     FIN_WAIT2
tcp 0 0             192.168.0.206:1883          192.168.0.204:49655     TIME_WAIT
tcp 0 0 192.168.0.206:47898 yw-in-f113.1e100.:https ESTABLISHED
tcp 0 0 192.168.0.206:41538 ya-in-f189.1e100.:https ESTABLISHED
tcp 0 0 192.168.0.206:1883 192.168.0.204:49657 TIME_WAIT
tcp 0 0 192.168.0.206:54231 yw-in-f100.1e100.:https ESTABLISHED
tcp 0 0 192.168.0.206:51248 yv-in-f113.1e100.:https ESTABLISHED
tcp 0 0             192.168.0.206:1883         192.168.0.202:51213 ESTABLISHED
tcp 0 0 192.168.0.206:1883 192.168.0.204:49654 TIME_WAIT
tcp 0 0 localhost:1883 localhost:58199 ESTABLISHED
tcp 0 0 192.168.0.206:1883 192.168.0.204:49656 TIME_WAIT
tcp 0 0 localhost:1883 localhost:56996 ESTABLISHED

Hmmm, 192.168.0.202 with an established tcp connection above, is my windows machine where I connected to mosquitto via TT3

[papa]

    Just executed sudo nmap -sS -O 127.0.0.1  & got

Starting Nmap 6.40 ( nmap.org ) at 2016-01-25 10:48 CST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000072s latency).
Not shown: 994 closed ports
PORT STATE SERVICE
139/tcp open netbios-ssn
445/tcp open microsoft-ds
631/tcp open ipp
5555/tcp open freeciv
8080/tcp open http-proxy
8443/tcp open https-alt

    executed sudo nmap --top-ports 20 localhost

Starting Nmap 6.40 ( nmap.org ) at 2016-01-25 11:11 CST
Nmap scan report for localhost (127.0.0.1)
 Host is up (0.000031s latency).
PORT STATE SERVICE
21/tcp closed ftp
22/tcp closed ssh
23/tcp closed telnet
25/tcp closed smtp
53/tcp closed domain
80/tcp closed http
110/tcp closed pop3
111/tcp closed rpcbind
135/tcp closed msrpc
139/tcp open netbios-ssn
143/tcp closed imap
443/tcp closed https
445/tcp open microsoft-ds
993/tcp closed imaps
995/tcp closed pop3s
1723/tcp closed pptp
3306/tcp closed mysql
3389/tcp closed ms-wbt-server
5900/tcp closed vnc
8080/tcp open http-proxy

Hmm, port 1883 is not listed as open OR closed.  What gives?

[papa]

    OTOH, sudo iptables -L gets
Chain INPUT (policy ACCEPT)
target        prot opt source     destination
ACCEPT tcp -- anywhere     anywhere tcp dpt:1883

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

[kmac]

Try sudo nmap -p1883 localhost
That should show port 1883 open.

[computourist]

If a server is listening on Localhost there is no guarantee that it is listening on the external IP-address. Configuration or IPtables could prevent external connections.
(so use nmap -p1883 192.168.0.206 instead, just to be sure the server is also mapped to the external address)


The next step would be:

the fact that Mosquitto is up & running on the external interface (checked by the MQTT client) would make me think something is wrong with your Arduino.

The fact that there is a TIME_WAIT condition on the session between gateway and broker indicates that there has been a session once, that was somehow interrupted.
This would suggest that ethernet hardware is probably OK..

I would concentrate on the Arduino and Ethernet shield and check wiring and software programming.

[papa]

kmac, sudo nmap 127.0.0.1 -p1883 produces:

Starting Nmap 6.40 ( nmap.org ) at 2016-01-25 13:03 CST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00015s latency).
PORT STATE SERVICE
1883/tcp open unknown

so that way shows 1883 open, but I'm surprised the service is unknown & not mosquitto

[papa]

computourist, running from Terminal on the openhab / mosquitto linux machine,

sudo nmap -p1883 192.168.0.206 produces

Starting Nmap 6.40 ( nmap.org ) at 2016-01-25 13:07 CST
Nmap scan report for 192.168.0.206
Host is up (0.00016s latency).
PORT STATE SERVICE
1883/tcp open unknown

I will recheck the Gateway hardware & software.

[kmac]

My output also shows 1883 as open/unknown.
I have no entries in IPtables on my mosquitto machine.
Maybe try removing yours and see what happens?

[papa]

Computourist & all, I'll repeat that this gateway (hardware & software 2.2) connected to another network (& to openhab /mqtt on windows) before I transported it elsewhere in a protected box. When I reprogrammed it, it was mainly changes to connect to my current location's network. CT, you well know that the gateway has a fairly simple schematic

That said I'm re-cheching & reporting the connections.

The base is a Buono Uno switch to 3.3 volts. For this gateway, I use a Seed Studio Ethernet Shield to which I soldered headers for the needed gateway connections. After the soldering, I continuity checked the connections. The Ethernet Shield is plugged on top the Buono.

      Via the Ethernet Shield, the gateway makes the following Arduino connections:

Arduino connection	Connected to:
D2	RFM69 DIO0
D7	MQTT LED positive via 100 ohm resistor
D8	RFM69 NSS
D9	Radio signal LED positive via 100 ohm resistor
D10	No connection
D11	RFM69 Mosi
D12	RFM69 Miso
D13	RFM69 Sck
GND by D13	RFM69 Gnd
3.3v	RFM69 3.3v
Gnd by 5v Gnd by 3.3v	Radio signal LED negative MQTT LED negative

  ^^ Note about LED resistors: computourist used 1 kilo ohm resistor with LEDs. I prefer 100ohm which lets the LED be somewhat brighter without burning out. For dimmer LED & to reserve more current to other node parts, use higher than 100 ohms. May 19, 2016, computourist said: With 100 ohms, current will be around 15 mAmps, well within specs.

From my checking all the connections are well seated in their sockets & I see no cross connections between parts & locations.

I've powered the Gateway via USB on both my windows & linux machines & via a wall adapter connected to a usb cable

   When Gateway was usb connected to the windows computer, I sometimes activated debug mode & checked the serial monitor & sometimes commented out debug mode.  For all the above, the gateway never connected to the gateway.

[papa]

kmac, sudo iptables -S produced
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -p tcp -m tcp --dport 1883 -j ACCEPT

then I executed sudo iptables -D INPUT -p tcp -m tcp --dport 1883 -j ACCEPT

Then nsudo iptables -S produced
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT

That is, rule for port 1883 was deleted, but Gateway & Mosquitto still do not connect

[papa]

OK, Computourist,

I started with a fresh copy of Gateway 2.2 & added my following customization:

//#define DEBUG // uncomment for debugging. I did it both ways, giving no difference

byte mqtt_server[] = { 192, 168, 0, 206}; // MQTT broker address
byte ip[] = { 192, 168, 0 , 212 }; // Gateway address (if DHCP fails)

//Match frequency to the hardware version of the radio (uncomment one):
//#define FREQUENCY RF69_433MHZ
//#define FREQUENCY RF69_868MHZ
#define FREQUENCY RF69_915MHZ

#define ENCRYPTKEY "my_keyxxxxxxxxxx"

const char *subTopic = "home/rfm_gw/sb/#"; // MQTT subscription topic ; direction is southbound.
const char *clientName = "RFM_gateway"; // MQTT system name of gateway.
char buff_topic[30]; // MQTT publish topic string
char buff_mess[32]; // MQTT publish message string
void mqtt_subs(char* topic, byte* payload, unsigned int length);

No errors or warnings with uploading, except low memory when uncommented debug mode

The Gateway still does not connect with MQTT

[papa]

Now I've installed the Ethernet Shield, etc on top another Buono Uno.

According to my router DHCP table, the Gateway gets an address, now 192.168.0.203

I rebooted the Linux computer & ran several updates.

When Updates finished, mqtt was running & I restarted openhab.  Gateway still not connecting to MQTT

I wonder why TT3 connects to mqtt on the Linux machine, but nothing from TT3 shows up on a subscribe to all messages in a Terminal

When I first run TT3, I can connect almost immediately with 192.168.0.206:1883.

But if I disconnect, I cannot reconnect without closing & reopening TT3  ?

[papa]

Maybe another clue toward getting Gateway connected to MQTT broker on linux

I just started Mosquitto from a Terminal prompt & got the following

$ sudo mosquitto
1453930879: Warning: Mosquitto should not be run as root/administrator.
1453930879: mosquitto version 0.15 (build date 2013-08-23 19:23:41+0000) starting
1453930879: Opening ipv4 listen socket on port 1883.
1453930879: Opening ipv6 listen socket on port 1883.
1453930879: New connection from 192.168.0.203.
1453930879: Invalid protocol "MQTT" in CONNECT from 192.168.0.203.
1453930879: Socket read error on client (null), disconnecting.

The last 2 lines keep repeating.

So looks like the Gateway keeps trying to connect to the MQTT broker then gets a socket read error & disconnects.

Now who can tell me how to fix this or where to look for fixing it? Thanks in advance.

[papa]

The error message in the last post was the clue I needed. I googled Invalid protocol "MQTT" in CONNECT

That took me to this article

esp8266hints.wordpress.com/2015/02/06/mosquitto-invalid-protocol-mqtt-in-connect-errors-and-esp8266-continuously-trying-to-reconnect-to-broker-service-without-success/

I had to adapt some things to my version of Linux (Linux Mint 17 based on Ubuntu 14.04) (will aim to document later)

        but after following the above instructions with some adaptations ...

I got some beautiful sights: The Gateway's MQTT LED lit up & serial monitor reported "got connection with MQTT server"
            On Linux Terminal I saw:
    New connection from 192.168.0.203 on port 1883
     New client connected from 192.168.0.203 as rfm_gateway

" Yaaaaaaaaaaaaaah Hooooooooooooooooo !  Now I can proceed with the rest of the test with Linux

As I had suspected, it looks like the Gateway was fine AND Mosquitto on Linux machine was almost fine, but needed something more so Gateway & MQTT could stay connected.