| Records | Load Time | Parse Time | Total Time | Column Sort |
|---|---|---|---|---|
| 10,000 | 114ms | 101ms | 215ms | < 1 sec |
| 100,000 | 278ms | 447ms | 725ms | ~ 1 sec |
| 1,000,000 | 1749ms | 7550ms | 9299ms | ~ 5-6 secs |
| Records | File Size |
|---|---|
| 10,000 | 909K |
| 100,000 | 9,083K |
| 1,000,000 | 90, 821K |
<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" layout="absolute" viewSourceURL="srcview/index.html">
<mx:Script>
<![CDATA[
import flash.utils.getTimer;
import mx.collections.ArrayCollection;
import mx.utils.StringUtil;
import mx.rpc.events.FaultEvent;
import mx.rpc.events.ResultEvent;
private var initialTime : int;
private var loadComplete : int;
private function loadData() : void
{
initialTime = getTimer();
httpService.url = source.text;
httpService.send();
loadButton.enabled = false;
}
private function onResult( event : ResultEvent ) : void
{
loadComplete = getTimer();
var result : Array = [];
var arr : Array = event.result.toString().split( "\n" );
for each ( var str : String in arr )
{
result.push( str.split( ", " ) );
}
dg.dataProvider = result;
loadButton.enabled = true;
resultText.text = "data loaded in " + (loadComplete - initialTime) + "ms, data parsed in " + (getTimer() - loadComplete) + "ms, total " + (getTimer() - initialTime);
}
private function onFault( event : FaultEvent ) : void
{
loadButton.enabled = true;
}
]]>
</mx:Script>
<mx:HTTPService
id="httpService"
result="onResult(event)"
fault="onFault(event)"
resultFormat="text"/>
<mx:ApplicationControlBar dock="true">
<mx:ComboBox id="source">
<mx:dataProvider>
<mx:Array>
<mx:String>assets/10_thousand.csv</mx:String>
<mx:String>assets/100_thousand.csv</mx:String>
<!--<mx:String>assets/1_million.csv</mx:String>-->
</mx:Array>
</mx:dataProvider>
</mx:ComboBox>
<mx:Button label="Load Data" click="loadData()" id="loadButton" />
<mx:Text text="" id="resultText" />
</mx:ApplicationControlBar>
<mx:DataGrid id="dg" bottom="10" top="10" left="10" right="10">
<mx:columns>
<mx:DataGridColumn dataField="0" />
<mx:DataGridColumn dataField="1" />
<mx:DataGridColumn dataField="2" />
<mx:DataGridColumn dataField="3" />
</mx:columns>
</mx:DataGrid>
</mx:Application>
We are currently working on a project using CSVLib in flex and have found out just how powerful Flex is for parsing large recordsets on the fly.
By the way, your example took my machine:
data loaded in 3314ms, data parsed in 36ms, total 3350
Thats a dual core 2.4Ghz with 4GB RAM on Vista 64bit
]]>Mac, Safari Flash Player 10
]]>I would be curious to see the difference in performance when the data is loaded using remoting. I would think it would be considerably faster.
I'm on an iMac 2.4, 2Gb ram:
data loaded in 14614ms, data parsed in 376ms, total 14990
Tim
]]>http://www.jasperpotts.com/blog/2007/11/faster-swing-lists-and-tables-upto-88000x/
Jasper
]]>I have a few comments / questions:
(1)
When you change the "dataProvider" property of DataGrid, is the rendering and updating synchronous or asynchronous? That is, does the call to "set dataProvider" block until the DataGrid rendering is complete? I'm a novice to Flex's thread model (I know there really isn't much of one) so I don't know. If the call is not synchronous - i.e. "set provider" returns right away, before the rendering is necessarily complete, then it might make sense to move the math from the final line of "onResult()" to a separate method called by the "updateComplete" event of the DataGrid. Something like this (leaving out enclosing "script" and "CDATA" pieces):
private function handleFinish(event:FlexEvent):void {
loadButton.enabled = true;
resultText.text = "data loaded in " + (loadComplete - initialTime) + "ms, data parsed in " + (getTimer() - loadComplete) + "ms, total " + (getTimer() - initialTime);
}
<mx:DataGrid id="dg" updateComplete="handleFinish(event)" . . . />
I tried this myself and couldn't really distinguish much of a time difference - so maybe "set dataProvider" really is synchronous. I dug into the Adobe source and found a call to "dispatchEvent()" but had trouble following from there. What do you think?
(2)
Just to clarify, are you saying that this:
private function onResult( event : ResultEvent ) : void
{
. . .
dg.dataProvider = result;
. . .
}
<mx:DataGrid id="dg" . . . />
Is faster than this:
private function onResult( event : ResultEvent ) : void
{
. . .
dg.myProvider.source = result;
. . .
}
<mx:DataGrid id="dg" dataProvider="myProvider". . . />
<mx:ArrayCollection id="myProvider" />
I'm having trouble proving one way or the other. My intuition would be to agree with you (one less layer of wrapping and change notification) but believe it or not, I sometimes get results where the second approach is faster. Again what do you think?
(3)
I think I got a minor improvement by substituting a String const for the ", " in the call to "split()".
Thanks again for this excellent, thought-provoking piece. This literally is what most Flex programmers writing business apps are thinking about every day, so I am very grateful you are addressing it!
Best,
-Noah
Also, I assume performance is a lot worse when we use XML, right? That's more to think about.
Thanks again for the article, it really got me thinking.
]]>2) You have to set the dataprovider to something first. You can't just set dg.dataprovider.source because by default, the dataprovider is null. You have to specify it as an instance of something. In this case I used "Array".
Collections are the preferred object to use instead of "plain-old" XMLList and Array because collections support bindings and include helper functions to easily update specific items in the collection. Basic Array or XMLList objects do not support binding or dispatch any data-binding events.
In this case, I did not want any binding events to fire b/c of the huge size of the dataset. Normally, you do want bindings to be applied, so that the grid contents are updated any time the contents of my collection is updated. The Collection classes are extremely helpful for keeping you view/data model in synch.
3)Thanks, did it speed it up much?
4) I imagine XML will be much slower b/c there is more complex parsing involved, and the language itself if more verbose, and would have a larger file size. I have not tested this.
]]> i have a csv file with date and other datatype column.
when i read it. it pick some date file and other field leave .
my dataset return all the record but some date field display empty.
how can i handled.
thanks
]]>Sorry I didn't reply sooner! My "item 3" sped things up only a tiny bit, really.
Thanks again for the article,
-Noah