Caution: Multiple Data Sends, Some Ethernet Shields

[lewishollow]

TL;DR - if your nodes are only sometimes succeeding when sending, try removing your southbound commands.

I just wasted a lot of time and energy debugging something, and I wanted to post about my experience in case anyone else runs into this...

I have a node on the gate to my front yard to let me know if the gate is open and also send back environmental data.  Fairly straightforward except that it runs off of four AA batteries that are charged via solar panels during the day, so I had to write a lot of code to try to make it go to sleep and only wake up to send when absolutely necessary.  All of that seemed to be working fine, except sometimes the state of the indicator about the gate state and the actual gate state would get out of sync (eg front gate is closed, indicator says it's open).

I brought the outdoor node inside and started debugging it.  It appeared that sometimes it would send fine, but sometimes it would not be able to send its state and eventually fail.  

I started with the easy stuff - resetting the components, checking all connections, no dice. I assumed I must have messed something up on the outdoor circuit and some solders were shorted or something.  

I spent hours taking it apart and back together again.  Still nothing.

I tried reprogramming my gateway node to make sure I had the w5100.h fixes.  In doing so, I somehow bricked nearly all of my Unos and burned a couple of days trying to bring them back to life to no avail.  I still have no idea what I did wrong there...

I finally got an Uno to take the updated code, and it STILL wasn't working reliably.

At the end of my rope, I started writing more complicated code to try to work around the issue so that, even if it got delayed, the message WOULD be sent.  Not ideal, but I didn't think I had anything else I could debug... until I looked at the lights on my gateway node.

I noticed that the Tx light was on constantly.  I looked at my openhab code - I was having it send WAY more southbound commands than I needed it to.  I didn't thinks this should matter, though, since the ethernet shield should be full duplex.  Once I removed all southbound sends, the gate node worked perfectly.  I felt like an idiot.  I was congesting the network.

I'm now in the process of slowly adding back nodes and checking the send status to make sure I'm not causing too much traffic.

I'm surprised, actually, since the full duplex indicator is on on the ethernet shield, so it should be able to send both ways just fine.  At some point, I'll debug through the gateway node more now that I know what I'm looking for.

ragoth - every time I sat down to work on this project, this is what I was working on. I get very little time to work on this stuff, so this literally took weeks to get through.  Now that I've figured this out, I'll have more time to get to your questions.  I'm sorry about the delays.

[papa]

Thanks, lewishollow, for sharing your experience & what has helped alleviate the problem. We often do learn more from the hard stuff than we do from the easy. "Tx light was on constantly" was a good clue to explore.

Just to clarify what you learned... You said, "I was having it [my openhab code] send WAY more southbound commands than I needed"
Please add some specifics: Were the sb commands in items or rules or both? Were there rules that fired on a frequent basis? How often?

[greginkansas]

I have seen the same thing to many sends will flood the network. I put a few waits in some rules to keep sends from being to close to each other. Also this send sub seems to let the nodes spread the traffic. I use 60sec time on my nodes. The delay(500) seems to move the node times.


void txRadio() // Transmits the 'mes'-struct to the gateway
{
digitalWrite(OnboardLED, HIGH);
retx = true;
int i = 0;

while (retx && i<6) {
if (radio.sendWithRetry(GATEWAYID, (const void*)(&mes), sizeof(mes),9, ACK_TIME)) {
retx = false;

} else delay(500);
i++;
}
//numtx = i; // store number of retransmissions needed
digitalWrite(OnboardLED, LOW);
} // end txRadio

[papa]

I'm hoping lewishollow will say more about "too many south bound sends."

Maybe my 11 nodes are not enough to conflict with each other (20-30 second txinterval).

I find my nodes' txintervals update the UI just fine mostly without rules to nudge them.

I try to use rules sparingly: To update a UI virtual switch only when the nodes' push button has changed the actuator state.
To set a timer to turn a node actuator on or off.

That said, Greg, the send subroutine you quoted above may come in handy if I suspect send conflicts later.

[lewishollow]

The sb rules were items that I was triggering via rules. Any time a node wasn't doing what I expected it to, I would try to fix it with openhab code changes instead having to unplug the node and bring it to my office to reprogram. For example, I have RGB LEDs on my nodes, and I treat each color as a different device in my code, each of which I would update in each operation, meaning that if I ever set it to blue, I would also send red off and green off to be safe, not thinking that that would just add more traffic. When the LEDs weren't reflecting what I expected, I would try to ameliorate the issue by making the lights update every few seconds instead of just when a change was detected, which, unbeknownst to me, was making things worse. Also, I'm not great with cron, so it's possible that I was actually sending messages a lot more frequently than I thought. To get actual numbers, I would have to check all of my cron expressions, which I'll be doing as I re-add my nodes. In the meantime, suffice to say that I was having the gateway send out messages to nodes that it didn't need to that, at least at this time, seemed to cause problems with my network traffic.

[papa]

Thanks, lewishollow, for the additional information on what happened. It confirms my decision to avoid rule sb sends unless a change in state warrants it.

[computourist]

Hi All,

I had simular problems as lewishollow . Most of the time all was OK, but sometimes my gateway would hang, and the transmission led would stay lit.
I did not change my openhab config, so no change in sb messages. First I thought it had something to do with reception quality, and wrote the retry code in the gateway.
But this didn't really help. At the same time here on the forum people were reporting uptimes of weeks and months, so I decided the code was not to blame.

I remembered the W5100 chip has a serious design flaw, and decided to follow up this path. The W5100 chip drives the MISO line constantly; it cannot float. This means another SPI device cannot drive MISO.
When connecting multiple devices (like in the gateway) this would cause conflicts. The remedy is an inverter that drives the SEN signal from the inverted SS signal. I'll attach the application note for reference.

W5100_AN_SPI.pdf (541.62 KB)


I knew the ethernet module I was using (see below) did not have this inverter. I installed this inverter, and since then my gateway has been working flawlessly. Root cause found, problem solved.

If you have problems with your gateway, it is wise to investigate whether this inverter is implemented on your ethernet board.

Early versions of the W5100 shield and separate W5100 modules (like I used) have this defect.

Arduino W5100 shields from version 2.0 and on have the inverter fitted, so no problem there. If in doubt, download the schematic of your ethernet shield and check.



My flawed ethernet module is shown below. If people are interested I can upload a picture of my repaired module.

[computourist]

Please remember the limited bandwidth used for communication over radio.
Default for the RFM69 library is 4800 b/s; max is 300 kb/s.

Since the gateway is a concentrator for traffic, the thruput of a single gateway is max 4800 b/sec, assuming perfect synchronisation of traffic..
Nodes are not synced, so collisions will occur causing a lowering of the effective thruput of the system.

In short: keep the number of messages per node limited, and transmission intervals long...

[greginkansas]

I spread the msgs for the wall display out like this.

rule "Send temp to wall display"
when
Time cron "25 0/11 * * * ?" // on minutes
then
Wall_guage_Temp.sendCommand(new DecimalType(node4temperature.state.toString))
// logInfo(" Wall_guage_Temp "," sending wall Temp" + Wall_guage_Temp.state)
end

rule "Send Hum to wall display"
when
Time cron "35 0/11 * * * ?" // on minutes
then
Wall_guage_Hum.sendCommand(new DecimalType(node4humidityFixed.state.toString))
// logInfo(" Wall_guage_Hum "," sending wall Hum" + Wall_guage_Hum.state)
end


rule "Send Presure to wall display"
when
Time cron "45 0/11 * * * ?" // on minutes
then
sendCommand(Wall_guage_Pressure, barometer_inHg_acurite_for_wall.state.toString)

// logInfo(" Wall_guage_Pres "," Org Presure " + barometer_inHg_acurite.state)
// logInfo(" Wall_guage_Proxy "," sending wall _Converted " + barometer_inHg_acurite_for_wall)
end

rule "Clear wall display zero" //1 hour15 minutes
when Time cron "7 15 0/12 * * ?" // on hours
then {
sendCommand(Wall_guage_Zero_guage, ON)
// logInfo("Wall_guage_Zero_guage "," Zero_guage")
} end

[lewishollow]

Thanks everyone. One thing I might do to my code base is make a queue to process a messages in a more network-friendly way with back-off times and checking for new northbound messages between each send. It might help prevent this in the future.

[computourist]

It may be easier to just increase bandwidth on the radio link.

[lewishollow]

May 24, 2016 18:55:04 GMT computourist said:

It may be easier to just increase bandwidth on the radio link.

Oh right... yeah.  I'll try that first.