Using OpenHAB Persistence (as for Charting)

[papa]

Using OpenHAB Persistence (as for Charting)

Until recently, in using OpenHAB persistence, I was largely shooting in the dark & borrowing from the OpenHAB demo. I’ve been trying some of it more carefully & learning from the results.

Without persistence, OpenHAB item data is rather fleeting or buried: It constantly updates on the User Interface & deep in the logs that record many OpenHAB details. As the name implies, “persistence” saves & persists OpenHAB information for later use. Three later uses for persistence that I can mention: 1) If OpenHAB is stopped & restarted, persistence can restore the previous item data to the User Interface rather than show undefined states until new data is suppled. 2) Persistence can save selected item data for the User Interface to show as a chart (a graph over time). 3) Persistence might save selected item data to be reported by other programs & methods as graphs, etc.

In this thread, I aim to document two available OpenHAB persistence methods: RRD4J & logging.  Caution: Persistence may store a lot of data, not a big issue for a regular computer (Windows or Linux). However, according to this, data writes can corrupt a Raspberry Pi’s SD card over time.

===========================================
        Addons Files needed for RRD4J & logging persistence:

In your computer’s addons folder, make sure you have these two files: org.openhab.persistence.logging-1.8.x.jar & org.openhab.persistence.rrd4j-1.8.x.jar.
    For Windows, copy the files to C:\openhab\addons
For Linux (including Raspberry Pi), execute these commands (giving your Linux password)
    sudo apt-get install openhab-addon-persistence-logging
    sudo apt-get install openhab-addon-persistence-rrd4j

===========================================

        openhab.cfg

No need to change the original in openhab.cfg for the sake of RRD4J or logging persistence??

==============================================

[papa]

Using OpenHAB Persistence (as for Charting), Continued

==============================================

Changes to persistence configs (rrd4j)

If needed, edit & re-save the rrd4j.persist file as the text below between ++..++ & ++..++. For Windows, go to C:\openhab\configurations\persistence. For Linux (including Raspberry Pi), go to /etc/openhab/configurations/persistence

Note: Much of the rrd4j.persist file is “commented out” or inactive due to “//” at the beginning of setting lines. The following line near the end is the new, active one:
* : strategy = everyMinute, restoreOnStartup

This thread illustrates using persistence, etc to display charted weather data on the OpenHAB User Interface, especially the posts at Oct 5, 2016 at 3:15pm & Oct 6, 2016 at 4:00pm

+++++++++++++++++++++++++++ text for rrd4j.persist follows to next ++..++
// Configuration file for "rrd4j" persistence module

// persistence strategies have a name and a definition and are referred to in the "Items" section

Strategies {

// for rrd charts, we need a cron strategy
everyMinute : "0 * * * * ?"
everyHour : "0 0 * * * ?"
everyDay : "0 0 0 * * ?"

// if no strategy is specified for an item entry below, the default list will be used
default = everyChange
}

/*
* Each line in this section defines for which item(s) which strategy(ies) should be applied.
* You can list single items, use "*" for all items or "groupitem*" for all members of a group
* item (excl. the group item itself).
*/
Items {
// To persist all items once a day & on every change & restore them from db at startup,
// UNcomment the next line
// * : strategy = everyChange, everyDay, restoreOnStartup

// To persist all weather values every hour, UNcomment the next line
// Weather_Chart* : strategy = everyHour

// To persist all temperature & weather values every hour, UNcomment the next line
// Temperature*, Weather* : strategy = everyHour

// to store EVERYTHING once a minute & restore them from data base at startup, uncomment the next line
* : strategy = everyMinute, restoreOnStartup
}
+++++++++++++++++++++++++++ text for rrd4j.persist is above to previous ++..++

Hint from here: for rrd4j persistence to work in OpenHAB charting after its configuration is changed, one may need to delete its data base (db) files so new db files are created.

    To expand the above hint … Stop OpenHAB. Find the folder that holds the .rrd files & delete all those .rrd files. On Windows, that folder may be at C:\openhab\etc\rrd4j For Linux (including Raspberry Pi), that folder is found at /var/lib/openhab/persistence/rrd4j

        Added Note: Your OpenHAB log may give an error like
...rrd4j.internal.RRD4jService] - Removing invalid definition component = null heartbeat = 0 min/max = 0.0/0.0 step = 0 0 archives(s) = [] 0 items(s) = []

According to this, don't worry about that error, especially if it happens only once per OpenHAB run.  Looks like it's just a hiccup at OpenHAB start up.

=========================

Note that RRD4J's data base storings happen no more frequently than every minute.  That's also the most frequently data points will show in a chart based on RRD4J.  Your OpenHAB set up may generate data more frequently, but how it's stored & charted can apparently be no more frequent than every minute.  (Of course, more frequent data stores would make the files much larger much faster.)

[papa]

Using OpenHAB Persistence (as for Charting), Continued

==============================================
Changes to persistence configs, (logging.persist)

As needed, edit & re-save the logging.persist file as below between ++..++ & ++..++. For Windows, go to C:\openhab\configurations\persistence. For Linux (including Raspberry Pi), go to /etc/openhab/configurations/persistence

Note: some of the logging.persist file is “commented out” or inactive due to “//” at the beginning of setting lines. The following line near the end is active:
Weather_Chart* -> "Weather"
If you do NOT have a Weather_Chart group in your items file, change that line to:
      // Weather_Chart* -> "Weather"
If you want / need to log items in a Sump_Chart group, add the line:
     Sump_Chart* -> "Sump"

+++++++++++++++++++++++++++ text for logging.persist is below to next ++..++
// Configuration file for "logging" persistence module

// log all changes of items listed in the Items { } section
default = everyChange
// in the file Weather.log in [your path]openhab\logs folder
// A time stamp begins each entry.
// In openhab.cfg, Weather Binding, Wunderground, updateInterval=5 so time stamps between an item’s entries will be about 5 minutes apart
// if you stop & restart OpenHAB, there will be a time gap in the entries

Items {
Weather_Chart* -> "Weather"
}
+++++++++++++++++++++++++++ text for logging.persist is above to previous ++..++
                ==========================

[papa]

Using OpenHAB Persistence (as for Charting), Continued

==========================================

Changes to sitemap configuration

This thread illustrates using persistence, etc to display charted weather data on the OpenHAB User Interface. See the post Oct 6, 2016 at 4:00pm for sitemap changes IF you want to display weather data.

In the Sump Monitor Node thread, search for "Add to sitemap file entries"  to see sitemap additions for charting Sump Monitor data.

[papa]

        For a given purpose, one OpenHAB persistence method may be better than another.

This community.openhab.org post describes advantages & disadvantages of various persistence methods & how to use more than one persistence method in our OpenHAB configurations.

[papa]

My elderly Linux laptop (12 years old) is one host for my OpenHAB & MQTT. Lately it was getting very sluggish & even bogging down to need a forcible shut down. I checked the folder of openhab logs ( /var/log/openhab2 ) & discovered over 100 MB of log files. Not necessarily a big deal on a bigger hard drive, but apparently was on this older small HD. I deleted all the backup, earlier log files that ended in .1, .2, etc. Old Linux laptop is back pretty spry again.

I also checked the folder for persistence files ( /var/lib/openhab2/persistence ).  Those files storing for rrd4j & mapdb methods are not very big, but we may need to keep an eye on their size.

    Update:  While the above may bear watching, I'm not so sure the above matters caused the bulk of the old laptop's sluggishness.

Of course this old machine does not have a lot of ram.  It has needed rebooting every so often from ram that probably gets clogged with memory usage that did not get reclaimed.

Also I had disabled a rules file while openhab2 was running (probably not a good idea). When the machine finally rebooted, the hard drive was tick-tick-ticking away.  I checked the running processes with top, showing that openhab was eating a lot of memory & cpu resources. I wonder if openhab was looking for that disabled rules file.

I stopped openhab, disabled the rules file, & restarted. Now openhab is more modest in its demands on the machine.

[jimkernsjr]

Hi Papa...

A couple points:

A bit nicer way to store the data in a way it can be used again is with MySQL. Article here on install:

community.openhab.org/t/openhab2-mysql-persistence-setup/15829/1

At the end, add PHPmyAdmin to access your database tables from the web browser.
sudo apt-get install phpmyadmin
then navigate to
http://ipofserver/phpmyadmin.

Works great is your server is LAMP (Linux/Apache/MySQL/PHP)already.

Also

Never a problem to change the rules "on the fly" - Openhab monitors the changed files for this, items, and sitemap - no need for a restart. At least on Linux, I don't know about Windoze nor care to try.

You can reduce the count of rules it keeps (default 10), as well as the size (10mb)...might be a good way to take it easier on your antique

location: installdir/userdata/etc
file: org.ops4j.pax.logging.cfg

entries to change under
# File appender - openhab.log and # File appender - events.log
log4j.appender.out.maxFileSize=10MB
log4j.appender.out.maxBackupIndex=10

PS...watch the disk space...Linux uses a journaling file system. You need to leave 10% to avoid fragmentation, which may be more your problem if the disk is THAT full. It recaims some on reboot...

I also determined I accidentally created a race condition in my rules...I had one blind changing the other's position, then when the second blind decided it changed, it told the first then....

Point is, sometimes if the logs are big, check them! Get rid of the java crashes (we've all seen them), and the size can be manageable. I can go a month at least without a rollover.

Regards,

[jimkernsjr]

And another quick point...I dont know if it matters, but when you complete the setup of the MySQL + OpenHAB configuration, go to the paper interface/configuration/system and set your default persistance.

After that, pretty data! (attached)

Attachments:

[papa]

Thanks for checking in on this, jim & supplementing my limited understanding of Linux. Turns out the antique Linux laptop's hard drive was getting flaky, working ok for a while, then tick tick ticking away & taking forever to do anything, sometimes flaking out completely.

I got all the files I needed from the Linux laptop & moved over to a Raspberry Pi running OpenHAB 1.x. It's more hassle to edit openhab files than on the laptop, but fortunately I don't have as much editing to do for a while.

I've had mixed results with some aspects of persistence, especially in using the extensions.

        Note:

To this thread, I moved jimkernsjr's post about automating irrigation.

To this thread I moved jimkernjr's post about using a windows computer to edit OpenHAB config files on a Raspberry Pi.