Home Forum Community URAD data

This topic contains 6 replies, has 4 voices, and was last updated by  sorin 6 months, 1 week ago.

Viewing 7 posts - 1 through 7 (of 7 total)
  • Author
    Posts
  • #6036

    sorin
    Participant

    Hello

    By any chance, is it possible to get URAD data for longer periods, ie. 1-3 years? Or, is it possible to get such data for a specific location in a numerical format?

    Thank you!
    Sorin

    #6039

    daniweb
    Participant

    WOuld it be possible to have the possibility to inject all the measures in a InfluxDb ?

    #6093

    Radu
    Keymaster

    @sorincheval the data older than 2 months is archived. You can get access to that but there will be a fee. Either as license or fixed amount. Contact me for more details.

    @daniele-wyss yes, but what’s the point to that?

    #6095

    daniweb
    Participant

    The combination of influxDb and grafana offer a very powerful graphics creation.
    For example look there where I put together info coming from KNX, RF OregonScientific and uRADMonitor A3 (local jason).

    #6391

    ecloud
    Participant

    A fee, are you kidding? I see this as a community project. The more you centralize or throttle or keep proprietary any aspect of it, the less useful it becomes to the world.

    Yes I know storage costs money. But some of us would be willing to host our own data. I’d like to use IPFS somehow, but not sure yet whether there is a database that is a good fit for this kind of time-series data on IPFS. Maybe will evaluate OrbitDB, but it’s written in JS so I’d have to run it on NodeJS… ick.

    I have experience with InfluxDB. I have doubts about its ability to scale, based on what I’ve heard at work (we use it for another purpose: automated software benchmarkes and test results and such). But on my system at home ATM it’s only taking up 157MB of memory, and I’ve been recording the data from one uRadMonitor at a time more or less continuously since February 2017, and a few other data points from other devices here and there. We’ll see how much it increases later. (I was just thinking that I should use influx to log its own memory usage, so that I can graph it and look for a trend.) My crappy little script to insert data (which I run from a cron job) is at https://github.com/ec1oud/uradmonitor-influxdb-inserter But I suspect if you try to store all the data worldwide it will be much more demanding.

    I used to use https://en.wikipedia.org/wiki/RRDtool (a round-robin database) for weather data years ago. It didn’t have performance problems (being designed a couple of decades ago for the computers we had back then), but you have to decide in advance how the data will be resampled or otherwise reduced as it ages. The idea is to have high-resolution data for the most recent time period and summarized data for longer periods. I’m not fond of the idea of losing any of this data too quickly; who knows how many years is really enough. Whereas with Influx you can just start storing data very easily, and change the “retention policy” later on when it gets too unwieldy.

    Anyway if you can’t afford the storage, it would be nice if you can provide a worldwide aggregated stream of some sort so that the rest of us can experiment with alternate means of storage. Perhaps a multicast channel would work, sending out a big JSON packet once per minute with all the readings from all the devices? It would not require much bandwidth.

    #6411

    Radu
    Keymaster

    Sadly I am not kidding. Yes, as you said, this is a community project but one that needs to evolve continuously. This includes finding all the support and resources for the endless list of things that need to be implemented. While for private individuals there’s a large openness towards providing free access, universities reasearch institutes and companies do have funds for acquiring such sets of data. And that is a good thing to take into account and use for the purposes mentioned. Feedback?

    #6412

    sorin
    Participant

    “While for private individuals there‚Äôs a large openness towards providing free access” – I asked the question about data availability as a private person, not as an institution. Following your latest answer, could I have free access to data? I intend to play with them e.g. to compare with other data and publish the results indicating the source of the data.

Viewing 7 posts - 1 through 7 (of 7 total)

You must be logged in to reply to this topic.