DIY Home Automation, Capabilities & Limits

[papa]

Responding to another thread, this thread topic came to mind as a poster reported persistent communication glitches in his system & computourist responded as follows:

Hi sam4205,  You seem to have some trouble when 'stress-testing' the gateway & endnode. Your findings are consistent with the way things are designed/set up.

Main requirement when starting this project was simplicity and duplex communication. It was designed as a data gathering / control network with limited real-time performance. When measuring temperature one or 2 seconds don't count...

The design is based on:
- Software (both gateway and end node) is designed around a simple loop.
- Interrupts are blocked in the gateway in order to share the SPI bus between 2 users: RFM69 and ethernet.
- Interrupts are not used to flag external events; the normal program flow is not interrupted.

This means that this design is prone to 'collisions' and will only function if:
- Events do not occur at the same time or in very short timing intervals.
- The amount of end nodes and messages that share a radio link (network ID) is limited
- Data transfer allows for delays, so retransmission is a viable option to increase reliability.

The 'loop' design is a simple setup and easy to understand/adapt; it means all processes take place one after the other.
This also means that when the gateway is talking to Mosquitto, it cannot receive or send data over the radio link and vice versa.
It also means that if an end-node is debouncing a switch or writing to an LCD it cannot send/receive data.

Not using interrupts in such a setup means you are bound to loose some data on the radio link. A retransmission scheme is used to deal with this. Since the gateway-radio can be offline for quite some time (when handling/decoding MQTT messages FE) it seemed wise to choose a rather long retransmission interval (.5 Seconds) that would lead to rather long response times in case of data collision. But I prefer a long delay over loosing data ;-)

In the end it all comes down to a tradeoff between:

- Simplicity of program setup
- Number of nodes/messages that need to be served
- Response time required.

Of course things could be improved a lot by choosing an interrupt driven , store/forward, message stack based design. You state you are a software person, so you might be able to improve ...

Given the current design you could try the following:

- Decrease the transmission retry interval. It is set at an arbitrary value of 0.5 seconds, and I have not experimented with that.
- Block radio retransmission by setting the number of reTx to 1. Error handling would need to be done in Openhab, but your response times will definitely improve.
- Examine the load on your MQTT broker. If MQTT response time is bad, your radio response times will suffer.
- Examine the software on your node. Keep the event-handling part as short as possible. If needed, offload event handling to external hardware or an extra Arduino.

            Some remarks on your [sam4205] post are inline below:

     yesterday at 3:21am sam4205 said:  Thanks Computourist, appreciate your inputs.

I agree with you about the expectations from this simple design and other ideas to build out a complex design. I am a software person so don't have lot of hardware knowledge

I really like the idea to let the node do most of the stuff and push the info instead of pull by gateway, as suggested by papa, greg and yourself. I have tried those tweaks and while troubleshooting, I found some basic issue in my setup related to long time needed for communications.

One of the post, I stumbled across, was on the LowPowerLabs forum, where Felix mentioned that typically messages should take less than 100 ms to communicate with reasonable strength (~-70db or more).

--> I agree, but the response time in this system is also dependent on performance of MQTT/Openhab: how soon is the gateway available again after handling MQTT messages. Many forum members run Mosquitto & Openhab on a Raspberri Pi with limited performance. Handling MQTT by the Pi is a factor to consider.

In my setup, I have push switches on the node that could be toggled either by push or via openhab. Leaving two toggles out of picture, and while trying just one switch at a time, I see issue in my design. Ack times is very variable and has wide ranges.

--> as explained above. Also: debouncing pushbuttons in software is NOT good for your performance...

I have attached the screen shot with MQTT log and message time from Serial.

- After wakeup, the nodes sends 0, 2, 17,18,19,20 and 48 status to gateway
- First line in the serial shows Dev19 took only 137 ms (highlighted in red)
- Dev20 took 941 ms
- Dev48 took 288 ms
- After few seconds, I toggled the Dev17 on openhab and it was immediately received by the node(my led on node changes immediately) and then
it took 5636 ms to send the ACK back to the gateway, with 4 retries. Between that 5636ms, the gateway reported connection lost to node.
- After few seconds, I toggled Dev18 on opehab and it was also immediately received by the node and my led toggled immediately. But ACK back to the gateway took 328 ms with 1 retry.
- The RF strength of -46db seems to be a strong strength, but still its taking longer time to send back the ACK.

--> As explained above: your signal strength is NOT the limiting factor, handling concurrent events is. The "connection lost" report is an immediate result of the gateway being busy with other things than radio reception.

I don't know if the issue is at the Node or at the Gateway. I have seen messages being sent in less than 100 ms, but only sometimes. Node and Gateway are just 20 ft away.

--> Again: signal strength is not the issue...

[papa]

greginkansas followed up to say: "... Everyone, nothing wrong with having two gateways if radio traffic is not what you want. ..."

       some of which echoed what I [papa] was thinking & writing at the same time to post:

Thanks, computourist, for the DIY Home Automation overview with respect to Sam's situation. You confirm & inform my sense that one needs to steward the demand on the system. Keeping in mind how many nodes are connected & how many functions on each. Choosing what's most important to know & accomplish. After prioritizing those, spreading the remaining demand on the system: Using rules to pull data only when absolutely necessary (& then not too many at any one time), but more often letting nodes push data. Using an overall node TXinterval (time between data sends) that is adequate & perhaps using a more immediate data send for one function. Not running or reporting too many functions from a node, unless those run infrequently. Accepting some transmission losses as long as OpenHAB stays relatively up to date.

computourist, you said "The 'loop' design is a simple setup and easy to understand/adapt." For someone like me that started with a little experience building & programming Arduino circuits, it was certainly not easy (getting parts & building nodes from a schematic, interlinking OpenHAB-MQTT-Sketch software, troubleshooting), but was doable with some digging. A more complex approach could be daunting for those who want to get their feet wet in DIY Home Automation.

Question, computourist: Could more than one gateway stretch the system's capability? How much? I suppose that would depend on the processing power of the computer hosting MQTT, right? A system built around a Pi would reach its limits sooner than a regular Linux or Windows computer? In which case, it should not be too hard or expensive to move a Pi-centered system to an older computer with Linux installed.

This discussion might be a good thread topic [& in fact is implemented in this very thread]: DIY Home Automation, Capabilities & Limits. Contributors could each list how many Gateways & nodes they have, TXintervals, functions on each node, etc. Where did we seem to bump against limits & how did we adjust?

[papa]

For this thread, I would appreciate many posts, which might be brief, but willingness to accept followup questions.

    Giving this kind of information for what's running in each of your systems:

___ number of Gateways (software version ____)

___ number of End nodes

Then please list a summary of what you have running on each node (including the Gateway). What hardware (sensors, buttons, relays, etc)?  What data does the node send & how often (TXinterval)?   What data does OpenHAB pull from the node using items & rules?   (If more than one node is the same, just tell the number & what all have. )

On what computer are you running OpenHAB & Mosquitto (MQTT)?  Raspberry Pi (which version)?  Regular Linux computer?  Windows computer?  Other (like computourist's Cubietruck)?

What limits are you bumping against?  Power/current limits? Delays?   Other limits discovered or suspected?  How have you addressed the limits & with what success?

I will start summarizing my system below & again would appreciate several others.

        Perhaps we can learn from each other & help us gain the most results from this approach

                & perhaps foresee brick walls that suggest taking another approach.

[papa]

                computourist added more:

        To answer a few questions from previous posts [emphasis by papa]:

@ papa : using more than one gateway certainly stretches the systems capabilities because it lowers the probability of a collision occurring. You could even have one gateway for low reliability (continuous temperature updates, state monitoring) and another for high reliability (only a few buttons that get operated maybe once an hour). In this way you could maybe miss or have a delay on a temperature message, but it gets updated once every few minutes anyhow. At the same time the probability of two push buttons being operated simultaneously in the other network is almost zero.

Having multiple gateways does not influence load on the MQTT broker: it has to serve an equal amount of messages, regardless whether they originate from one, two, or more gateways.

I started out running Mosquitto and Openhab on a Pi (the first one) and found it somewhat sluggish. I switched to a Cubietruck for my home server and am satisfied with its performance (wifi access point, webserver, mosquitto, openhab, squeezebox, Cups print server, Samba file server).

@ greginkansas : about [version 2.5]: In my opinion the capabilities of the 2.x setup are more or less reached. A new system would mean a major upgrade (3.0?) to include the techniques I described in earlier posts [see above]. This is a major undertaking for which I presently lack the time. On the other hand: my current system is meeting my needs perfectly...

@ sam4205 : I understand you have a need for multiple push buttons on a single node. Maybe I can find the time to try some latches to improve the design..

Thanks again, computourist, for all your efforts & provided information & help.

[papa]

            greginkansas responded:

I'm at 8 nodes sending once a minute and just got 2 more with half having to send BTN or PIR. I think that might be about it for 95% real time performance. The PI2 has Openhab, Mosquitto, squeezebox server and the Echo bridge server and runs 2-15% load so its good for another gateway. I think the load on the gateway is the limit.  -Greg

[papa]

My Current DIY Home Automation

Windows 7 laptop runs OpenHAB & Mosquitto (MQTT)

I have OpenHAB regularly checking an NTP time server & yahoo weather (via http binding)

One Gateway running software version 2.2 (I have not yet felt the need to upgrade yet & be able to see how many message retransmissions are required, but may need to when I add more nodes)

The Gateway has MQTT & Radio LEDs & one jumper switch whose state is reported to OpenHAB & then some rules operate on the basis of that status. (The switch state ON indicates a "security" mode where nodes with solid state relays turn on lights pseudo randomly around certain times.0

        In the past, I have had up to 11 end nodes & things seemed to work fine.
Gateway & end nodes are built on Buono Uno Arduino compatibles, switch to (mostly) 3.3 volt operation.

7 end nodes presently, TXinterval (time between data sends) = 20 seconds

For all 7, OpenHAB items get their RSSI & Voltage via MQTT messages

6 nodes have DHT11 sensors & OpenHAB items read their temp & humidity via MQTT

5 nodes have push buttons, Solid State Relays (SSR) & OpenHAB switch items toggle the SSRs & rules make sure use of nodes' push buttons are recorded in the switch items. Depending on the state of the Gateway's "security" switch, other rules can pseudo-randomly use these nodes to toggle SSRs / lights.

[Arrrrrrrrrgh, proboards can find the most inconvenient time to log one out & lose a long write ]

Besides DHT11, push button, & SSR, one of the above nodes has an infrared LED & sketch code that can toggle a TV on / off via an OpenHAB switch item or a rule.

Besides reading RSSI & voltage, another of the above nodes (garage) uses an Arduino digital pin to trigger a DC to DC SSR connected (via several feet of wire) to a garage door opener wall switch terminals. An OpenHAB switch item can control this.

Besides reading RSSI, voltage, & DHT temp / humidity, another garage node uses a reed switch to sense if the garage door is completely closed or even partly open. An OpenHAB contact item reads / displays this state.

When I had 11 end nodes in this system, most were the basic computourist DHT end node with push button, DHT11, & LED output. Some were battery powered. Via MQTT all sent RSSI, voltage, temp, humidity, & output state. Some nodes used a PIR sensor & reported that to OpenHAB.

When things started working (with usual fits & starts), things mostly worked well.

        Limits I experienced & how I coped:

I did not get ONE garage node to read RSSI, voltage, temp & humidity, reed switch state, AND reliably toggle the garage door opener via an Arduino digital pin & a DC to DC SSR. When I pulsed the SSR, reading the reed switch state became unreliable, suggesting I bumped power / current limitation. To cope: I used TWO garage nodes. Both sent RSSI & voltage. One also sent temp, humidity, and reed switch state. The other just pulsed the SSR with a digital pin. This works reliably.

I built a node to send RSSI, voltage, temp, humidity, push button state, & toggles a DC to AC SSR (the kind in a box with screw terminals & LED). The Arduino digital pin could toggle the SSR's LED, but did NOT trigger AC current to flow. Instead I use the digital pin output to trigger a transistor to allow the Buono Uno's FIVE volt DC to flow & fully trigger the SSR. Apparently this provides enough power & works reliably.

Added:  One of the above is a light switch node that sends RSSI, voltage, DHT11 temp & humidity, push button state, & toggles a DC to AC SSR (chip package with pin terminals).  Since these SSRs take much less power to trigger, the Arduino digital pin output alone is enough.

In another geographical location, I have another small OpenHAB system running, just one Gateway & one DHT end node.

Sometimes I've run OpenHAB / MQTT on an older laptop with Linux installed. Other times, I've used a Raspberry Pi 2.

Eventually to my main system, I intend to add more nodes & functions & perhaps another Gateway.

I look forward to hearing your reports of where you run OpenHAB / MQTT,
how many gateways & nodes, & how much you ask each part of your system to do.
What limits are you encountering & how do you address them. Thanks.

[papa]

sam4205 reminded me of this thread about Raspberry Pi's performance related to OpenHAB.

On that thread, some people report robust systems running on Raspberry Pi 2, but discourage using Pi 1

One major suggestion was to avoid major writes to the Pi SD card because the cards corrupt sooner.  Rather for writes, use a USB drive of some kind.

Visit the link for more details.

[papa]

Now coming available is the Raspberry Pi 3, still $35.00,supposedly 50% faster than the 2 (TEN times faster than a 1) & with built in wifi & bluetooth (not yet supported in the software).
        Form factor is backward compatible to the 2, except LEDs moved.  Will have upgraded power source, 2.4 amps to handle higher built in demand & what might be plugged in.

Here's a couple overviews: this one & this one.  Here's a lot of preview if you want.

The 3 should be even more capable for OpenHAB, DIY Home Automation.

[greginkansas]

And the Pine64. Kai is making a openhab image for it.

[papa]

Pine64, interesting. I did not know it until you posted it, greg.

[chrisinkc]

Here is my setup:

Pi 2 running MQTT and OpenHAB 

I am running Gateway 2.3

Garage Node 1) DHT11, Gas sensor, Light Sensor, 2 reed switches (one for each garage door).  I originally had a flame sensor on this node, but it was always acting buggy, so I removed it.
Garage Node 2) Only 1 DC to DC SSR to open garage door 1
Garage Node 3) Only 1 DC to DC SSR to open garage door 2
Basement Node 4) DHT11, Gas sensor, flame sensor, 4 light sensors.

These have been working well except one of my SSRs was firing off multiple times for a couple of weeks.  Its now working just fine and I did nothing to fix it so I don't know what was going on with it.

Just last night I decided to tap into the old ADT system that was installed when my house was built (prior owner) but that I have never used.  All of the doors have built in switches (I assume reed switches??) to monitor the doors.  So, I pulled out the old ADT board and installed a node to take advantage of the door switches.  This became node 5.

Basement Node 5) 1 light sensor and 5 reed switches.

All of my nodes are Arduino Buono Unos w/ 433mhz radios.  All nodes are using lewishollow's base code

I am now having some strange behavior from my door sensors (Node 5 only).  Sometimes, but not always, when you open a door, instead of getting one notification that the door is open, I get like 50 notifications in rapid fire.  The notifications go back and forth between open and closed.  When I close the door, sometimes, but not always, it will rapidly fire off a bunch of notifications again.  Sometimes the sensors work just fine.  Since these are simple reed switches (I'm assuming...) I don't see how it possible that the switch is somehow being physically switched on and off that fast.  I know my code is good because I used the same code that I used for my garage reed switches and they work perfectly.  I'm thinking that maybe I am maxing out either this particular node or maybe I am maxing out my system???

Here are my questions:

1. Am I maxing out my basement node 5 with one light sensor and 5 reed switches?
2. Am I maxing out my Pi2?  I just got a Pi3.  Should I upgrade my setup to a PI3?
3. Am I to the point that I need to add another gateway?
4. Has anybody else run into this issue that I am seeing with my door switches?

[greginkansas]

Are you debouncing the reeds? The main loop might be running so fast you get rapid fire msgs.
I do not think you are maxing out the system.
Do you have strong pullup/pulldowns?
These are mine now and get fast lights on from PIR
PIR>NODE>gateway>openhab>Inseton hub> light
lots of hops and its fast

Node 2 Changed at Saturday, 08:40:36
Driveway Changed at Saturday, 08:41:12
Node 6 Changed at Saturday, 08:40:59
Node 8 Changed at Saturday, 08:40:44
Node10 Changed at Saturday, 08:40:42
Node 12 Changed at Saturday, 08:40:32
Node14 Changed at Saturday, 08:41:00
Node16 Changed at Saturday, 08:41:16
Node18 Changed at Saturday, 08:41:08
Node20 Changed at Saturday, 08:40:52

[computourist]

Always best to use a software blocking period to prevent too many messages flooding the network. In my DIG node I use a 10 second blocking period.

[chrisinkc]

This is what I have in my Arduino sketch for each reed switch:

#ifdef REEDSWITCH3ENABLED
reedSwitch3Data = new DigitalInputDataClass(REEDSWITCH3DEVICEID, REEDSWITCH3PIN, 10);
reedSwitch3Data->periodicSendEnabled = true;
#endif //REEDSWITCH3ENABLED

I'm assuming the "10" is denoting the 10 second blocking period, correct?

I am using 10k resistors for each switch. Same as I used on the garage doors which work perfectly fine.

[chrisinkc]

Well, I figured out my problem! Turns out I had a little bit of solder touching multiple pins on my board with all of the resistors. As soon as I cleaned it up, everything started working perfectly.

[papa]

Yes, Chris, that's why I like to continuity test (with a multimeter) along with my soldering, making sure that I've made the right connections & avoided making the wrong connections.

             You had said earlier ...

Here are my questions:  Am I maxing out my basement node 5 with one light sensor and 5 reed switches?
     Am I maxing out my Pi2? I just got a Pi3. Should I upgrade my setup to a PI3?
          Am I to the point that I need to add another gateway?

Then you said, "As soon as I cleaned it [too much solder] up, everything started working perfectly."

    Does this mean that you answered your above questions with "No, I have not maxed out the system's capability [the theme of this thread]"  ?

[papa]

My Current DIY Home Automation
(updated 5/22/2016)

Windows 7 laptop runs OpenHAB (1.8.1, 1.8.2 does not automatically update the UI) & Mosquitto (MQTT)

I have OpenHAB regularly checking an NTP time server, but yahoo weather (via http binding) is not working

One Gateway running software version 2.2 (I have not yet felt the need to upgrade yet & be able to see how many message retransmissions are required, but may need to when I add more nodes).  Sometimes, I trade in a Gateway 2.4.

The Gateway 2.2 has MQTT & Radio LEDs & one jumper switch whose state is reported to OpenHAB & then some rules operate on the basis of that status. (The switch state ON indicates a "security" mode where nodes with solid state relays turn on lights pseudo randomly around certain times.0

In the past & now in the present, I have had up to 11 end nodes so far & things seemed to work fine.
Gateway & end nodes are built on Buono Uno Arduino compatibles, switched to (mostly) 3.3 volt operation.  (Five volts is still available at one Arduino socket & at several other locations if a shield is added.

I presently have 11 end nodes running.  Their TXinterval (time between data sends) is from 20 to 30 seconds.

For all 11, OpenHAB items get their RSSI & Voltage via MQTT messages

7 nodes have DHT11 sensors & OpenHAB items read their temp & humidity via MQTT  This gives me readings for my basement, main floor, upper floor, & garage.

6 nodes have push buttons, Solid State Relays (SSR) & OpenHAB switch items toggle the SSRs & rules make sure use of nodes' push buttons are recorded in the switch items. Depending on the state of the Gateway's "security" switch, other rules can pseudo-randomly use these nodes to toggle SSRs / lights.

Besides DHT11, push button, & SSR, one of the above nodes has an infrared LED & sketch code that can toggle a TV on / off via an OpenHAB switch item or a rule.  I detected the sent code & adapted a library to send it.  The node can still send the code, but I've moved the TV off site.

Besides reading RSSI & voltage, another of the above nodes (garage) uses an Arduino digital pin to trigger a DC to DC SSR connected (via several feet of wire) to a garage door opener wall switch terminals. An OpenHAB switch item can control this.

Besides reading RSSI, voltage, & DHT temp / humidity, another garage node uses a reed switch to sense if the garage door is completely closed or even partly open.  It also uses a simple "switch" of two bared wire ends & the end of a metal dead bolt to sense if the dead bolt is locked on a people entrance door.  OpenHAB contact items read / display these states.

Besides reading RSSI & voltage, another two of the above nodes sense if the dead bolt is locked on a people entrance door.

Added 5/22/2016:  One of the above is now a "Freezer" Node that senses temp & humidity outside a freezer (or other enclosure or liquid) AND senses temps inside a freezer (or other enclosure or liquid).

Some of my nodes have been battery powered. Some nodes have used a PIR sensor & reported that to OpenHAB.

When things started working (with usual fits & starts), things mostly worked well.

        Limits I experienced & how I coped:

I did not get ONE garage node to read RSSI, voltage, temp & humidity, reed switch state, AND reliably toggle the garage door opener via an Arduino digital pin & a DC to DC SSR. When I pulsed the SSR, reading the reed switch state became unreliable, suggesting I bumped power / current limitation. To cope: I used TWO garage nodes. Both send RSSI & voltage. One also sends temp, humidity, reed switch state & dead bolt state. The other just pulses the SSR with a digital pin. This works reliably.

I built a node to send RSSI, voltage, temp, humidity, push button state, & toggles a DC to AC SSR (the kind in a box with screw terminals & LED). The Arduino digital pin could toggle the SSR's LED, but did NOT trigger AC current to flow. Instead I use the digital pin output to trigger a transistor to allow the Buono Uno's FIVE volt DC to flow & fully trigger the SSR. Apparently this provides enough power & works reliably.

Added: One of the above is a light switch node that sends RSSI, voltage, DHT11 temp & humidity, push button state, & toggles a DC to AC SSR (chip package with pin terminals). Since these SSRs take much less power to trigger, the Arduino digital pin output alone is enough.

For a while in another geographical location, I had another small OpenHAB system running, just one Gateway & one DHT end node.

Sometimes I've run OpenHAB / MQTT on an older laptop with Linux installed. Other times, I've used a Raspberry Pi 2.

To my main system, I'm adding more nodes & functions & perhaps another Gateway (if necessary).  From time to time, I will update this thread accordingly.

I look forward to hearing your reports of where you run OpenHAB / MQTT,
how many gateways & nodes, & how much you ask each part of your system to do.
What limits are you encountering & how do you address them?  Thanks.

[papa]

Back when in the thread, Star vs mesh topology, we discussed using more than one Gateway & RFM69 network to spread the communication load & extend the capability of this DIY Home Automation project. However, until lately I had not seen or tried to work out all needed to have more than one Gateway/RFM69 network connected to the same MQTT server & OpenHAB UI.

I just figured that out & posted the documentation in the thread Using More Than One Gateway/RFM69 Network

Update, Dec. 16, 2018:  See this post for links to sketches that add the option of multiple RFM69 gateways / networks.