Stubborn MQTT gateway

[danychouinard]

Greeting everyone

I have been impressed about 2 years ago about the instructable explaining how to build an home automation system with Arduino and OpenHab. I have since try to build my own system. It's been quit a long time and I'm about to give up. I really need help.

At first I ordered some random part only to find out they were not compatible. Those Uno are so easy to find. As I was reading I found how to use them anyway and I ended up with level converter and even those fancy card from adafruit with level converter build in. Still no success.

Time to order proper parts don't you think ? I believe I have everything as require. Nothing to could then go wrong now :



My gears

So in order :

- AMS1117 5V (4.5-7V) Turn To 3.3V DC-DC Step down Power Supply

- RFM69HW 20dBm RF FSK transceiver 915 mhz

- Pro Mini Atmega328 3.3V 8Mhz

- Mini W5100 LAN Ethernet Shield Network Board Module R3 W5100 3.3V Compatible



So there again, not trying to outsmart anything.




My wiring


So let's try to build a fresh environnement for compiling.


A fresh install of Arduino 1.8.4-windows.

Then some code : computourist/RFM69-MQTT-client/Gateway_2.5/

Then some library :

• github.com/LowPowerLab/RFM69
• github.com/LowPowerLab/SPIFlash
• github.com/knolleary/pubsubclient

And some esoteric fiddling because noody like to be interrupted.


C:\Program Files (x86)\Arduino\libraries\Ethernet\src\utility\w5100.h

Line 344 and line 345 now looks like that :

inline static void setSS() { cli(); PORTB &= ~_BV(2); };

inline static void resetSS() { PORTB |= _BV(2); sei();};

Of course, Ethernet.h have to actually use the change, so uncomment line 5 and tell where the file is :

#include "utility\w5100.h"



It then compile, as they say on my cereal box, just right. I've selected the board "Arduino pro, pro mini" and the processor "ATMEGA 328 ( 3.3v , 8 MHz )"


It goes down the drain right about here.


Some adjustement :

Enable debugging ( line 70) of RFM_MQTT_GW_25



My Ethernet values :

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0x00, 0x01 }; // MAC address for ethernet

byte mqtt_server[] = { 192, 168, 11, 10}; // MQTT broker address (Mosquitto)

byte ip[] = { 192, 168, 11 , 11 }; // Gateway address (if DHCP fails)


my frequency and encryption key

#define FREQUENCY RF69_915MHZ

#define ENCRYPTKEY "nope"


And compile for glory.


The serial port tels me :


GW Version GW v2.5

915 Mhz



after a while :

No DHCP

connecting

no link

no link

no link



and so on.


Funny thing, If I unconnect the wireless card, the Ethernet card start to work fine.

GW Version GW V2.5

915 Mhz...
connecting
no link
MQTT-link OK

and I can even ping it from my computer.

I have tried so many permutation. Checked, doubled check and heptaquinta check my wires that my eyes are metaphoricly bleeding. I'm totally out of idea and really really need help.

[papa]

Welcome, danychouinard, sorry you are having trouble. As I say often, getting a working gateway & first node working & communicating with each other is the hardest part. About two years ago, I had the same situation. It tends to get easier after the first gateway & node.  It is a thrill when it works, but early stages can be "stubborn."

Thanks for all the details & humor you provided. (You've covered a lot of what I might ask or suggest.) It looks like you are NOT using the Uber instructable, which is good. What this forum uses is modified & more reliable (once it works) than the instructable. Some people have luck with the level shifters, but I'm skeptical about that approach.

From the parts you are using, it looks like you may be following computourist's guidance which should be good. You have maybe seen in my threads that I use different parts (Buono Uno Arduino compatible & Ethernet Shield installed on top the Arduino).  But again I believe you now have parts like what has worked for computourist & others.

I may have more to say later & perhaps ask more questions.

For now, I mention one quick thing that might help. It looks like you are powering the RFM69's VCC with the pro mini vcc (thru the breadboard's power rail).  Also you give a good clue: " If I unconnect the wireless card (I assume you mean the RFM69 radio), the Ethernet card start to work fine."

In computourist's schematic (which also uses a pro mini), one can see that he did not connect RFM69's power pin (VCC or 3.3 volts) to the pro mini's VCC pin. Instead he connected a 3.3 volt power supply directly the the RFM69's VCC / 3.3 volt pin. Why? The RFM69 radio (especially the one with "H" or higher power in its name) requires a lot of current when it is communicating. A pro mini cannot reliably supply that so the circuit gets unstable.

So you might try: disconnect RFM69's VCC from pro mini VCC. Instead connect RFM69's VCC directly to the 3.3 volt output of the AMS1117 power supply. Connect the AMS1117's 3.3 volt positive output to the breadboard's positive rail.  Then to the breadboard's power rail, connect BOTH RFM69's vcc AND pro mini vcc.  Connect the AMS1117 GND to the breadboard's negative rail. Then you can connect RFM69 GND & pro mini GND to the breadboard negative rail.

So trying powering the RFM69 differently & then post your Serial Monitor results.

[danychouinard]

Thanks for your answer Papa.  It's really nice to feel supported.

I think I've omitted to post how I power my cards ( and mostly the RFM which is actually the High powered one ).  




image uploader


So my lower power row is 5v'ish to power the ethernet card and the top row is gettting my nifty 3.3v that goes to the RFM69 and the Arduino mini pro VCC pin.  I beleive it is the correct way to plugged things up.  I however been wrong in the past, or so I've been told.  I've tested it by disconnecting the serial 3V3 pin ( did you know that the serial connection keep working just fine that way - neet ).

The power supply can provide 1500A - the best use I had with my radio satellite receiver as per now.

But you know what.  You made me look again at the schematic and I found out that the pro mini had his two ground pin grounded.  Could it be possible that...

[danychouinard]

I actually got excited about that

Well, pretty much more of the same for me.



I know however that if I disconnect the RFM69 MOSI pin the ethernet card start to work fine.

I have to leave now for few days to work.  I'll pounder more about that next week.

Good week y'all

[papa]

OK, thanks for explaining how you power the cards. That may be correct.

I suggest you make sure the W5100 is sending 3.3 volt signals to RFM69's MOSI, MISO, DIO0
(Disconnect the W5100 board from the bread board except for power & gnd. Connect multi-meter (MM) common terminal to GND. Then connect MOSI to the muti-meter's other terminal. On the MM, watch for the voltage as the W5100 flashes a signal. Then test MISO.

danychouinard: "if I disconnect the RFM69 MOSI pin the ethernet card start to work fine"

papa: Do you mean that the Ethernet card works fine if ONLY the RFM69 MOSI pin is disconnected & the rest of the RFM69 pins are connected? <- If this is true, make sure you are connecting RFM69 MOSI to the right place (D11). Also using multi-meter's continuity test option, make sure there is a good connection between the RFM69 MOSI solder pad & the end of the wire.

Could the RFM69 be defective? Do you have another RFM69 to try?

After you've put the project aside for a while, with fresh eyes, compare your wiring to the schematic.

[danychouinard]

Thanks again for your help.

I have about 10 RF cards and plenty of arduino of all sorts.  My middle name is "the part swapper".  The only part that I lack is the ethernet part.  You only need so many, right.  The one I'm using rigth now is the replacement one.

I'm pretty sure stuff working at 3.3v.  The SCK line is at about 1.4v, wich I found strang but since it is plugged as the schematic, I don't think I can't do much about it.

Here is the MOSI line.




At this point I doubt that it is ever be goint to work.  Reading the forum make be beleive that some people have done it but it feel somehow from another planet.  I'm even afraid to put more time and money in that project by bying the ESP8266 just to make a fool of myself again.  I mean, I should just be following schematic right ?  It's not like it is a complexe projet.  Thats why I was excited about.  A couple wire and a beautiful design.  It was just poesie.


Anyway, unless I can find more assembly instruction and parts specification that are known to work.  I will have to give up.

[papa]

Yes, I understand your reluctance to continue, danychouinard. No, it's not super complex, but it has enough details that even little things can cause problems. As I keep saying, getting a working gateway & first node communicating with each other is the most challenging part. I don't remember clearly how long that took me in the beginning, but it may have been at least a couple weeks of hitting it hard. Since then I've built 3 gateways & a couple dozen nodes & they usually work fairly soon.

I feel like building a gateway with a Buono Uno & an Ethernet Shield on top & an RFM69 radio installed in the shield requires the least amount of wires & soldering & provides the right voltage & current without an extra power supply. However, forum members (including me) have also had bumpy starts with that.

When you wrote, "if I disconnect the RFM69 MOSI pin the ethernet card start to work fine" it seemed like you might be getting close to success. When you say, "The SCK line is at about 1.4v" I'm not sure what that is supposed to be & will have to check.

After my beginning struggles, I've had much success with the RFM69 approach. For sensors AND for combined sensors & AC appliance control, I feel RFM69 is best. However, struggles like yours caused me recently to recommend the ESP82xx approach as the place to start in this project (after I experienced ESP82xx for myself) & then add RFM69 for sensors.

With ESP8266, once you have OpenHAP & MQTT installed on the computer you only need one device. You can use one recommended Itead Sonoff, for a modest amount of money. You can program the Sonoff with no soldering. Carefully connect an appliance cord to the Sonoff. The Sonoff should then connect with your WiFi (maybe after you reboot WiFi & Sonoff a couple times) With the needed entries in the OpenHAB config files, you should be able to control the appliance. ESP82xx threads by jimkernsjr & me could help you do this.

Of course, it's up to you. You might for now, set aside the RFM69 stuff (still constructed). Try one Sonoff on OpenHAB. Then come back to the RFM69 approach with fresh eyes especially if you want to use sensors. Recheck how you connect the Ethernet Gateway & the RFM69 radio to the Arduino, including RFM69's NSS connected to Arduino D8.

[danychouinard]

I'm not as good at giving up either I beleive. I've notice that you are recommending the Buono Uno and I was for once able to look it up on Aliexpress ( Buono Uno search without more specification just gives a list of disturbing figurine ). So there is my newest shopping cart :

www.aliexpress.com/item/BUONO-UNO-R3-Compatible-Arduino-UNO-R3-Improved-3-3V-5V-Selectable-Cable/32603451084.html
www.aliexpress.com/item/1pcs-Arduino-Shield-Ethernet-Shield-W5100-R3-UNO-Mega-2560-1280-328-UNR-R3-W5100-Development/32709442003.html

I believe that This would let me build both projets. If anyone had a better link pointing to a better hardware ( at Aliexpress ), I'm all hears.

I will try to avoid Sonoff as I think it will not help me to accomplish my goals. Hopefully I'll on day ends up with quit a bunch and sensors all over the place, over two houses.

But first thing first, let's complete the gateway.

[papa]

I admire your perseverance with the RFM69 approach, danychouinard. That's what finally got me & others to success.

As to your shopping list, you can also search the Buono Uno & Ethernet Shield on eBay. That's where I've bought mine.

When you get your parts, I recommend you carefully follow my Building... thread. I tried to make that thread as step by step & easy as possible.

Hint about the Buono Unos: They have a small switch to use 3.3 volt mode. Sometimes one must jiggle the switch a bit so that the Buono is powered & lights its power LED.

With the Buono Uno & Ethernet Shield, one just plugs them together, only one way to do it. No long wires to add resistance. Less wires to troubleshoot. In fact, with those parts (& even the LEDs), one can build the gateway without a breadboard. No need for a specialized power supply.  Just 5 volts supplied to the Buono Uno's USB connector (via computer USB or phone charger).

In building the gateway, just be sure to disable the VCC pin in the Buono Uno's ICSP cluster. Otherwise you'll still send 5 volts to your RFM69 radio. (The Building... thread has a link to posts on building the gateway & those posts cover such things.)  Keep your RFM69 wires fairly short.  Mine are about 38 mm.
    ===============================================================

While you wait for parts, I keep thinking about what you said (& the supporting Serial Monitor results): "if I disconnect the RFM69 MOSI pin the ethernet card start to work fine"  That suggests to me: 
Your Ethernet card is probably wired correctly to the Pro Mini. However, things are somehow not functioning correctly with the RFM69: not connected to all of the right Arduino pins? defective RFM69?  RFM69 not getting enough current? (wires too long, too much resistance)?  something else??

I'm not sure what to think of "The SCK line is at about 1.4v" I don't know & cannot find what SCK (clock signal) voltage should be, but I'd guess 3.3 volts. I do know the clock signal & timing is important to RFM69 function.  Maybe another indicator the RFM69 is not getting enough power so cannot operate reliably & then causes problems with the Ethernet card.

About RFM69 timing, see this (& the nearby diagram of SPI signal flow) at the end of here: "SPI is Synchronous - That means that the clock signal accompanies the data signal. MOSI - Master Out Slave In; MISO - Master In Slave Out; SCK - Clock signal from master to slave; SS - Slave Select signal selects slave devices. SPI is a bus and can have multiple devices on the bus. Because of a clock signal present, SPI can be operated faster than UART."

Please check another thing:  First, the 5v/3.3v power supply from the breadboard & disconnect the serial connection from the Pro Mini.  Then put a multi-meter (MM) in continuity mode.  On 5 volt side of the breadboard, put one MM probe at the 0 (zero) end of the GND (-) rail & the other MM probe at the 60 end of the GND (-).  Does the multi-meter make a "beep" sound indicating the GND (-) rail is continuous & connected from zero to 60?  Check the GND (-) rail on the 5 volt side of the breadboard.

If the GND rails are not continuous from zero to 60, do not do anything about it until I can caution you about some things.

[danychouinard]

Good evening

Well, I did receive the parts some weeke ago and I've confirm this weekend proper communication. Well, I had a small glitch between pin 2 and pin A02 but nothing serious.

The AMS1117 that I used to pull the voltage down drown a 9v battery within few hours of the DHT end node. Trying the HT1117 I found out that I did not have any 47 uF capacitor. Back to AliExpress and I'll be back in a month or so...

Just enough time to understant the mosquitto_pub syntaxe I guess ( could not get that dan led to shut off... )

Anyway, talk to you soon guys.


P.S. Thanks to you again Papa!

[papa]

Good to hear from you, danychouinard.

I'll be much tied up for the next 3 weeks, but I'll do my best to check in & try to give at least short responses.

I did not know that you planned battery power for the node. A 9 volt battery is compact, but you waste almost 6 volts to reach 3.3 volts.  3 AA or AAA batteries (alkaline or rechargeable?) might be better. A 3.7 volt lithium battery reduced a little via a germanium diode would be an option, but more money in the short run at least.  Be careful of buying cheap 3.7v lithiums.  In my experience, they may turn out to be fake junk.

        There are ways to decrease the node's power usage.

Look at activating the SLEEPY mode: Between data sends, the node goes into a low power mode. A watch dog timer wakes it after a pre-set lapse of time.

Also instead of connecting an always on 3.3 volts to DHT's VCC, you can connect DHT's VCC to a spare data pin. Set up that data pin for output & turn that pin off usually. Then when the microprocessor wakes up, power the data pin, give a delay for the DHT to stabilize, then read DHT data & send it.

Again, I don't have much time, but look at the following which I believe will do the above lower power operations.

   Try this program sketch & customize as you probably have before, except ALSO at line 181, UNcomment #define SLEEPY

Line 226 sets D7 as the pin to power DHT sensor (this could change to another available digital pin.)

Line 474 loads the library needed for sleeping the processor & gives the link for dowloading it.
Lines 1348-1364 handles powering the DHT with the designated power pin & giving a delay to settle the sensor for valid reading, & then reads the sensor using Celsius or Fahrenheit.
Lines 1393-1397 turns off power to the DHT

Lines 1957-1975 handle sleeping the processor & waking it up after the set delay
Line 206, the variable Loops sets how many loops of 8 seconds that the processor sleeps. (At present set to 8 times 8 seconds = 1 minute plus, but you can change it)

I would try this first on a node connected to USB & Arduino IDE Serial Monitor, then when it's working go to battery power.