I have a large data set (log data) that is stored in a SQLite3 database. I created a Python GUI using Wx.listctrl that displays that data from a query. It works fine for small amounts of data but obviously hangs on larger sets.. therefore I would like to implement one of two things:
-paging were the first 5000 records are shown in the listctrl and then if a user clicks a button the next 5000 are shown .. so on and so on.. OR -as the user scrolls down the listctrl its constantly populating with new data..
How would I implement one of these concepts? Thanks in advance for the help!
Make it a virtual control as described in the docs. The listview will invoke a callback as required in order to request the specific rows being displayed.
In different scenarios, the strategies will be different. The assumption is that you just use wxPython for presentation purpose, and you have a separate module/package dealing with the data. wxPython controls get data only from the data abstraction layer.
Scenario I: load all the log data into the memory and use wx.listctrl or wx.listview to display the lines.
Scenario II: fetch data from the database when the wx view is updated.
In Scenario I, you just need to setup an variable to remember the position the UI has read up to, or even you can simply use list sliding if you load the logs as list. Automatically updating the view by scrolling down/up can be implemented in scenario I because all data has been loaded and no more sqlite db connection will be made. It will be a bad choice to implemented auto-updating by scrolling down/up in scenario II since there would be too many db operations - imagine you scrolling down very quickly expecting to get the 5000th line?
In scenario II, you can setup a page size and use LIMIT/OFFSET to get the data set you want. You keep another variable for the offset number. With the variables page_size and offset, you can simply fetch the data and update the wx view. This scenario is especially good for limiting the memory use. I don't know how big your log is, but after all it is not a good idea to load all of it into the memory because a log keeps growing and you may also need to load other relatively large dataset at the same time.
