piGateway returns " SIGSEGV, Segmentation fault"

[acekrystal]

During my stability test I'm running multiple Odroids with piGateway, directly attached to the RFM69-433MHz chip.
The Odroid with the most modules registered (6 registered, from which 5 are online and each module pushing 4 payloads every ~3 seconds).
This Odroid is the only one that seems to crash at random times.

[33] to [1] Received Node ID = 33 Device ID = 4 Time = 376423012  RSSI = -57 var2 = 0.000000 var3 = 0.000000
Pinging node 34 - ACK - nothing![34] to [1] Received Node ID = 34 Device ID = 4 Time = 376935555  RSSI = -62 var2 = 0.000000 var3 = 0.000000
[31] to [1] Received Node ID = 31 Device ID = 4 Time = 171386087  RSSI = -60 var2 = 0.000000 var3 = 0.000000
[32] to [1] Received Node ID = 32 Device ID = 4 Time = 171949873  RSSI = -55 var2 = 0.000000 var3 = 0.000000
Pinging node 35 - ACK - nothing![35] to [1] Received Node ID = 35 Device ID = 4 Time = 360081860  RSSI = -62 var2 = 0.000000 var3 = 0.000000
[33] to [1] Received Node ID = 33 Device ID = 6 Time = 376424445  RSSI = -56 var2 = 21.296257 var3 = 55.461967
[34] to [1] Received Node ID = 34 Device ID = 6 Time = 376936986  RSSI = -62 var2 = 21.189438 var3 = 56.055542
Pinging node 31 - ACK - nothing![31] to [1] Received Node ID = 31 Device ID = 6 Time = 171387521  RSSI = -58 var2 = 21.322960 var3 = 55.428398
[32] to [1] Received Node ID = 32 Device ID = 6 Time = 171951310  RSSI = -55 var2 = 21.288246 var3 = 57.035172
[35] to [1] Received Node ID = 35 Device ID = 6 Time = 360083290  RSSI = -61 var2 = 21.293587 var3 = 55.187305
Pinging node 33 - ACK - nothing![33] to [1] Received Node ID = 33 Device ID = 4 Time = 376425562  RSSI = -57 var2 = 0.000000 var3 = 0.000000
[34] to [1] Received Node ID = 34 Device ID = 4 Time = 376938106  RSSI = -63 var2 = 0.000000 var3 = 0.000000
[31] to [1] Received Node ID = 31 Device ID = 4 Time = 171388641  RSSI = -57 var2 = 0.000000 var3 = 0.000000
[0] to [0] Invalid payload received, not matching Payload struct! 189 - 16
0000 f3 de 20 4b f0 8a 3b ea 2e 1a b3 c9 de c3 6d 72  ...K..;.......mr
0010 91 8c ec 2a c8 e6 7f a2 d7 d5 4f 37 cf 6d 49 49  ...*......O7.mII
0020 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49  IIIIIIIIIIIIIIII
0030 49 49 49 49 49 49 49 49 49 49 49 49 49 01 49 00  IIIIIIIIIIIII.I.
0040 c7 ff 09 00 02 00 00 bd b6 00 00 00 f6 5f 61 62  ............._ab
0050 00 f0 eb b6 90 a3 01 00 50 ee ff be 75 90 00 00  ........P...u...
0060 00 00 00 00 71 62 fe b6 44 f0 ff be 01 00 00 00  ....qb..D.......
0070 00 00 00 00 52 46 4d 2f 31 30 30 2f 23 00 00 00  ....RFM/100/#...
0080 01 00 00 00 78 a8 ff b6 f0 8b de b6 a4 7b de b6  ....x........{..
0090 b0 ee ff be 00 00 00 00 00 00 00 00 30 aa ff b6  ............0...
00a0 b9 a6 63 94 78 a8 ff b6 ff ff ff ff 98 ed ff b6  ..c.x...........
00b0 b4 f3 fd b6 68 f5 ff b6 14 45 ff b6 bc           ....h....E...

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb6bd7450 (LWP 8408)]
0x0000accc in RFM69::isr0() ()
(gdb) bt full
#0  0x0000accc in RFM69::isr0() ()
No symbol table info available.
#1  0xb6fb6aaa in interruptHandler () from /usr/local/lib/libwiringPi.so
No symbol table info available.
#2  0xb6c76fa0 in start_thread (arg=0xb6bd7450) at pthread_create.c:314
       pd = 0xb6bd7450
       unwind_buf = {cancel_jmp_buf = {{jmp_buf = {-1229097368, -1229097904,
               1, -1229099440, -1229099120, -1228361316, -1229097340,
               -1090523864, -221028936, -223781739, 0 <repeats 54 times>},
             mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0},
           data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
       not_first_call = 0
       pagesize_m1 = <optimized out>
       sp = <optimized out>
       freesize = <optimized out>
       __PRETTY_FUNCTION__ = "start_thread"
#3  0xb6e77d9c in ?? ()
   at ../ports/sysdeps/unix/sysv/linux/arm/nptl/../clone.S:92
  from /lib/arm-linux-gnueabihf/libc.so.6
No locals.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)

I think it has to do with incorrect handling of faulty payloads. (caused by bad signal, or collision of 2 or more payloads).
This might also be somewhat related to my other threat " homeautomation.proboards.com/thread/185/pigateway-returns-stack-smashing-detected ".

At an other Odroid I'm watching the interrupt function, there I also noticed that it is triggered a lot. So I'm kinda hoping this whole problem is caused by a not correctly working interrupt pin. It might be the interrupt pin is also floating and also needs a pull down resistor to keep it from randomly triggering.

This is the second time this happened to this odroid, first time was within 3 hour. I will first try the pull down technique with a 1K resistor
Will post my updates!

[acekrystal]

I just enabled the the logging of interrupts on this Odroid and later added the 1K pulldown to see if it got rid of the many interruptHandler function calls, but it didn't:

This is how my log looks like when uncommenting line 443 in rfm69.cpp

 printf("interruptHandler %d\n", intCount);

Result in my log:

[35] to [1] Received Node ID = 35 Device ID = 4 Time = 7768205  RSSI = -65 var2 = 0.000000 var3 = 0.000000
interruptHandler 1
interruptHandler 1
interruptHandler 1
[32] to [1] Received Node ID = 32 Device ID = 4 Time = 7783522  RSSI = -50 var2 = 0.000000 var3 = 0.000000
interruptHandler 1
interruptHandler 1
interruptHandler 1
interruptHandler 1
interruptHandler 1
Pinging node 31 - ACK - nothing![31] to [1] Received Node ID = 31 Device ID = 4 Time = 7757561  RSSI = -52 var2 = 0.000000 var3 = 0.000000
interruptHandler 1
interruptHandler 1
[34] to [1] Received Node ID = 34 Device ID = 4 Time = 20862769  RSSI = -59 var2 = 0.000000 var3 = 0.000000
interruptHandler 1
interruptHandler 1
[33] to [1] Received Node ID = 28017 Device ID = 6 Time = 719974234  RSSI = -56 var2 = 23.755630 var3 = 56.137943
0000 71 6d 06 00 5a ef e9 2a 88 0b be 41 41 8d 60 42  qm..Z..*...AA.`B
interruptHandler 1
interruptHandler 1
interruptHandler 1
interruptHandler 1
interruptHandler 1
interruptHandler 1
Pinging node 35 - ACK - nothing![35] to [1] Received Node ID = 35 Device ID = 6 Time = 7769679  RSSI = -65 var2 = 26.428627 var3 = 48.142216
interruptHandler 1
interruptHandler 1
[32] to [1] Received Node ID = 32 Device ID = 6 Time = 7785000  RSSI = -50 var2 = 23.413826 var3 = 58.698406
interruptHandler 1
interruptHandler 1
[31] to [1] Received Node ID = 31 Device ID = 6 Time = 7758991  RSSI = -54 var2 = 23.053329 var3 = 58.231480
interruptHandler 1
interruptHandler 1
interruptHandler 1
interruptHandler 1
interruptHandler 1

So to me it seems the "interruptHandler" function is called uppon a lot more then should be.
As I have added the 1K pulldown resistor to the interrupt pin I don't think this is caused by a floating pin anymore.
Also, if I unplug the interrupt pin the script is directly stuck and also doesn't come to live anymore when reconnecting the interrupt pin.

Questions to others:
Can someone with a RFM69-arduino-piGateway or RFM69-arduino-arduino-piGateway construction check if the interrruptHandler function is also called so often for them? The same happens to my other Odroids with only 2 sensor nodes.

[computourist]

Hi acekrystal,

Seems to me your interrupt handler is not working properly. Some thoughts:

- The RFM69 generates a lot of interrupts. To my knowledge, every time a packet is sent the RFM interrupts on completion of the send request. Also on every packet reception an interrupt is generated. This would explain the number of interrupts you see, so they are probably not caused by hardware problems.
- In another post you mentioned "stack smashing" to occur. IMO this is due to unproper handling of interrupts. On every interrupt data is pushed on the stack, and pulled again when the interrupt is handled. With simultanious incoming messages an interrupt can occur during the interrupt handling. This would interfere with the stack mechanism and lead to stack problems. Because in this situation the stack is not cleared properly this leads to memory leaks and eventually to segmentation errors.
- A convenient way to prevent this from happening is to mask interrupts altogether in the interrupt handling routine. This would lead to loss of data, but retransmission mechanisms will take care of that. Of course, this is easier done on a microcontroller like the Arduino than on a system with an Operating System running, like the Raspi.
- Hence the preference for running the gateway on Arduino ;-)

rgds,
Computourist

[acekrystal]

Thanks computourist,

I'm going to see if I can find a solution.
For my home I have it now running stable for about a month, with 2 modules that post 1 result every second.

But when I do that same at my office with 9, or even 5 modules, all pushing every second, the script doesn't run longer then 2 day's top. with 9 modules it often crashes within a hour.

At the moment I think I'm first going to look for a backup plan, like:
- a watchdog that looks if the script is still running ok, if not, restart the script.
- also the modules should not post this often. I'm looking more at 1 result every 5 minutes. Only the PIR should post every time it triggers, but in the beginning I can get away with storing this up to 5 minutes.


But I'm also looking into long term solution:
- looking into way's to handle the interrupt properly, I have been reading something about writing kernel modules or drivers that can do this and shouldn't be to hard to get into.
- A other solution I'm thinking about is to not giving the modules the power to uncontrollably shout when ever they want, but to let the Gateway manager when a module may push its next payload.
I'm not sure if the last is a realistic approach, but I'm curious what you think about it. I really would like to run these modules of battery's in the future, and I also would like to be able to change the measurement interval of these modules dynamically from a set interval time in the database. So this should perfectly match a system where the gateway try's to evenly spread all the measurements in the time window it has. I could also let the gateway poll the next results from all the modules one by one. Though, this should only be a nice addition to have less collisions, but it still doesn't fix a collision when it still happens.

[computourist]

Hi acekrystal,

nice te hear you are making progress.
The scheme you propose to avoid collisions has been used before. A long time ago IBM didn't trust the collision possibility on ethernet, so they came with the "token ring" mechanism.
A token is passed around nodes. Only when a node is in posession of the token it is allowed to send data. When finished, it passes the token to the next in line.
As we know all Token Ring networks have now been replaced by ethernet, but the mechanism is sound.
Have a look into it if you want to design a collision avoidance scheme....

Rgds,
Computourist

Oct 17, 2016 19:49:25 GMT acekrystal said:

Thanks computourist,

I'm going to see if I can find a solution.
For my home I have it now running stable for about a month, with 2 modules that post 1 result every second.

But when I do that same at my office with 9, or even 5 modules, all pushing every second, the script doesn't run longer then 2 day's top. with 9 modules it often crashes within a hour.

At the moment I think I'm first going to look for a backup plan, like:
- a watchdog that looks if the script is still running ok, if not, restart the script.
- also the modules should not post this often. I'm looking more at 1 result every 5 minutes. Only the PIR should post every time it triggers, but in the beginning I can get away with storing this up to 5 minutes.


But I'm also looking into long term solution:
- looking into way's to handle the interrupt properly, I have been reading something about writing kernel modules or drivers that can do this and shouldn't be to hard to get into.
- A other solution I'm thinking about is to not giving the modules the power to uncontrollably shout when ever they want, but to let the Gateway manager when a module may push its next payload.
I'm not sure if the last is a realistic approach, but I'm curious what you think about it. I really would like to run these modules of battery's in the future, and I also would like to be able to change the measurement interval of these modules dynamically from a set interval time in the database. So this should perfectly match a system where the gateway try's to evenly spread all the measurements in the time window it has. I could also let the gateway poll the next results from all the modules one by one. Though, this should only be a nice addition to have less collisions, but it still doesn't fix a collision when it still happens.

[acekrystal]

Oh computourist !

WoW, that is it!
lets use ethernet between te RFM69 and Odroid/RBPi. lets see.. I need a extra arduino for that.. with ethernet shield.... ow... wait....

ghihi