September 11, 2018 at 10:05 am #6036sorinParticipant
By any chance, is it possible to get URAD data for longer periods, ie. 1-3 years? Or, is it possible to get such data for a specific location in a numerical format?
SorinSeptember 14, 2018 at 10:13 am #6039daniwebParticipant
WOuld it be possible to have the possibility to inject all the measures in a InfluxDb ?October 3, 2018 at 9:34 pm #6093RaduKeymasterOctober 4, 2018 at 7:15 am #6095daniwebParticipant
The combination of influxDb and grafana offer a very powerful graphics creation.
For example look there where I put together info coming from KNX, RF OregonScientific and uRADMonitor A3 (local jason).
Attachments:January 6, 2019 at 10:52 pm #6391ecloudParticipant
A fee, are you kidding? I see this as a community project. The more you centralize or throttle or keep proprietary any aspect of it, the less useful it becomes to the world.
Yes I know storage costs money. But some of us would be willing to host our own data. I’d like to use IPFS somehow, but not sure yet whether there is a database that is a good fit for this kind of time-series data on IPFS. Maybe will evaluate OrbitDB, but it’s written in JS so I’d have to run it on NodeJS… ick.
I have experience with InfluxDB. I have doubts about its ability to scale, based on what I’ve heard at work (we use it for another purpose: automated software benchmarkes and test results and such). But on my system at home ATM it’s only taking up 157MB of memory, and I’ve been recording the data from one uRadMonitor at a time more or less continuously since February 2017, and a few other data points from other devices here and there. We’ll see how much it increases later. (I was just thinking that I should use influx to log its own memory usage, so that I can graph it and look for a trend.) My crappy little script to insert data (which I run from a cron job) is at https://github.com/ec1oud/uradmonitor-influxdb-inserter But I suspect if you try to store all the data worldwide it will be much more demanding.
I used to use https://en.wikipedia.org/wiki/RRDtool (a round-robin database) for weather data years ago. It didn’t have performance problems (being designed a couple of decades ago for the computers we had back then), but you have to decide in advance how the data will be resampled or otherwise reduced as it ages. The idea is to have high-resolution data for the most recent time period and summarized data for longer periods. I’m not fond of the idea of losing any of this data too quickly; who knows how many years is really enough. Whereas with Influx you can just start storing data very easily, and change the “retention policy” later on when it gets too unwieldy.
Anyway if you can’t afford the storage, it would be nice if you can provide a worldwide aggregated stream of some sort so that the rest of us can experiment with alternate means of storage. Perhaps a multicast channel would work, sending out a big JSON packet once per minute with all the readings from all the devices? It would not require much bandwidth.January 9, 2019 at 2:49 pm #6411RaduKeymaster
Sadly I am not kidding. Yes, as you said, this is a community project but one that needs to evolve continuously. This includes finding all the support and resources for the endless list of things that need to be implemented. While for private individuals there’s a large openness towards providing free access, universities reasearch institutes and companies do have funds for acquiring such sets of data. And that is a good thing to take into account and use for the purposes mentioned. Feedback?January 9, 2019 at 5:29 pm #6412sorinParticipant
“While for private individuals there’s a large openness towards providing free access” – I asked the question about data availability as a private person, not as an institution. Following your latest answer, could I have free access to data? I intend to play with them e.g. to compare with other data and publish the results indicating the source of the data.July 16, 2019 at 7:46 pm #6678JohanParticipant
You are paying a fee for the logging archive Radu keeps. I don’t agree with it completely either but you are free to log the data yourself for indefinite periods.
I grab the CPM data from a model A and publish it to a mqtt topic. Pimatic automagically makes a graph of it. But you could just as well scrape the webpage of any of the Uradmonitor products and log or broadcast it in any way you want, CSV, mqtt, database…
I do find it a bit odd that the logged data on the website is declared to be under some copyright. That is like saying Sony owns the copyright to all the pictures I made with my A7ii camera.
I don’t mind that archives are kept and made accessible for a fee but I can also withhold the data by firewalling the the units I own and there will likely not be any consequence for that.
- You must be logged in to reply to this topic.