Home > Development > blogs
Yesterday I had the fortune of reading a very interesting blog post by Thomas Jung showing how SAP's Web Dynpro ABAP can load and sort as many as 1 million records! It's a cool demonstration of the power of that platform. I'm not sure if Web Dynpro is a RIA platform or a fat client - it works with web browser and has a desktop runtime so perhaps its a "fit client" like Curl or Adobe AIR. Thomas was apparently inspired by Episode #17 of RIA Weekly in which I discussed the need for enterprise RIA platforms to be able to handle hundreds of thousands of records. He wanted to see how Web Dynpro stood up to that challenge and from what I can tell it did really well.
I decided to mess around with Curl to see how much data it could really handle. In a fit of curiosity I implemented some benchmark tests that are so unscientific they would make Descartes roll over in his grave. I ran the tests on my MacBook Pro on Windows XP through VMWare Fusion. Yes you read that right - not exactly the most preformant configuration in the world. Despite my less than stellar runtime enviroment I was really pleased with Curl's performance.
First I loaded up 10,000 records into a table. The table was a CSV file. Each record represents a contact and has five columns (i.e. full name, email, work-phone, home-phone). To get to 10k records I wrote a Java program that generates the records using UUIDs that are converted into strings for values of each field. I trimmed the string values to what I thought were appropriate sizes for various fields (e.g. 20 characters for name, 13 for phone numbers). You can get the Java program here.
I ran an example PIM program that comes with Curl which was already able to read a CSV file and load it into a Curl visual table widget. The Curl application launched and loaded the 10,000 records quickly. It was able to sort on any of the fields - none of which are indexed - instantly. Scrolling through the records was also instant - no delays at all.
Next I generated a CSV file with 100,000 rows of random records. Curl launched and loaded these records in about 4 seconds. It was able to sort on any column in about 2 seconds and provided instant scrolling through all 100,000 records.
Next I generated a CSV file with 1 million records. Curl took about 2 minutes to launch and load and about 25 seconds to sort on any given column. Surprisingly scanning through the 1 million records by moving or clicking on the vertical scroll bar was instantaneous. I have no idea why you can scroll through 1 million records with no latency but sorting them takes 25 seconds.
Finally I met my match when I generated a CSV file with 10 million records. The Curl application never launched thus ending my experiment - I actually fell a sleep waiting for the Curl application to finish loading, which it never did. I'm not sure what part of the system got hung up. It may have been Curl, or the Firefox browser, or the funky MacPro/Windows XP/VMware Fusion configuration I'm using. It's a probably a combination of all of these things.
I would love to see how other people's favorite platforms handle 1 million or even 10 million records. It looks like Web Dynapro is the current leader but I think Curl may have given it a better run for its money on a faster configuration. How many records can your favorite RIA platform handle?
- comments: 6
- curl
- dynpro
- performance
- sap
- tags:
Comments
6 Comments
Leave a comment
Stay Connected
Search InsideRIA
Poll: ECMAScript Reaction
The ECMA organization recently decided to stop work on ECMAScript 4 and begin a new version, tentatively described as ES "Harmony." How would you like to see this affect the evolution of ActionScript?
News & Events
Latest Features
Tag Cloud
- .net
- 360Flex
- 3d
- actionscript
- adam flater
- adobe
- air
- air api
- air cookbook
- air resources
- air xml webkit
- ajax
- ajaxworld
- alertthingy
- amp
- analytics
- ant
- antipattern
- apache
- api
- app engine
- apple
- applications
- april fools day
- aptana
- architecture
- asp
- audio
- authors
- away3d
- aws
- backward-compatablity
- bad usability calendar
- benchmarking
- berlin
- beta
- bitmaps
- blazeds
- blendmode
- book
- browserplus
- bsd
- business
- businessweek
- cairngorm
- camp
- career
- cdn
- chart
- cloud
- code
- code generation
- coldfusion
- command design pattern
- communication
- community
- compiler
- component
- compression
- conference
- conferences
- contest
- creativecommons
- cs3
- cs4
- curl
- customer
- cuzillion
- dare
- data
- datatable
- datavisualization
- dbunit
- ddlutils
- debug
- decompiler
- deep zoom
- degrafa
- design
- design patterns
- desktop
- development
- DHH
- diary
- dionalmaer
- drawing api
- dreamweaver
- drupal
- dynpro
- ebay
- ecmascript
- effectiveui
- ejb 3.0
- elixir
- embedded jboss
- encryption
- enterprise
- europe
- event
- excerpt
- experience
- ext
- externalinterface
- eyetracking
- features
- filters
- firefox
- fireworks
- fitc
- flash
- flash10
- flashforward
- flashplayer
- flex
- flex4
- flexbuilder
- flexmdi
- flexunit
- form factor
- framework
- friendfeed
- gadget
- gears
- geolocation
- gesture
- googledocs
- googlegears
- gpl
- graphics
- graphics mask
- guidelines
- guimark
- hosting
- how-to
- html
- ia
- ie
- ie8
- igoogle
- interaction
- interface
- interview
- iphone
- iphonedevcamp
- iron man
- j2ee
- jakob
- java
- javafx
- javaone
- javascript
- javase
- javase6
- jaxer
- jms
- job
- joyent
- keynote
- languages
- lcds
- learning
- lffs
- lib
- library
- license
- licensing
- linux
- listen
- livecycle data services
- location
- madrid
- map
- marketing
- math
- maturity
- maven
- max
- memory
- merapi
- mesh
- metrics
- microsoft
- mikepotter
- mit
- mix08
- mobile
- mockup
- model locator design pattern
- mono
- mozilla
- mvc
- mxml
- mxmlc
- neilson
- netbeans
- nio
- nitro
- nitrolm
- nokia
- objectivec
- offline
- omnigraffle
- onairbustour
- onairtour
- onairtoureurope
- ontologies
- oop
- open source
- openflux
- openscreen
- opensource
- optimization
- orm
- papervision3d
- paris
- pattern
- pattern mining
- patterns
- performance
- photoshop
- php
- podcast
- popular
- portable
- preload
- pricing
- prism
- Processing
- Processing.js
- profiler
- protagonize
- push
- qt
- rails
- release
- research
- resources
- ria
- ria community
- ria roundup
- ribbit
- rich
- ruby
- rules
- sanctuary
- sap
- scalenine
- scheduling
- scott sheridan
- search
- security
- seo
- server
- service oriented architecture
- services
- session bean
- silverlight
- singularity
- social
- sockets
- software
- source
- spain
- sproutcore
- spry
- sqlite
- static code
- stencils
- steveounders
- stockholm
- streaming
- survey
- swc
- swf
- switchboard
- syscon
- test driven development
- testing
- testng
- thermo
- tip
- tool
- tools
- training
- trends
- trolltech
- tufte
- tutorial
- ui
- ui pattern library design
- uirc
- unconference
- unit testing
- usability
- user experience
- ux
- vector images
- video
- view source
- vision
- voip
- warsaw
- wdn08
- web
- web 2.0
- web 2.0 expo
- webkit
- whenisgood
- why
- widgets
- windows
- workshop
- wwdc
- xhtml
- xulrunner
- yahoo
- yui
Related Books
Development Series
Get an overview of the tools and technologies that work together to allow developers to build Rich Internet Applications (RIAs) quickly and easily.
Recent Comments
- User Richard Monson-Haefel on The "One Million Records" Challenge: Hi Charles, Thanks for commenting on th...
- User Charles Kendrick on The "One Million Records" Challenge: We've had "stress tests" like these for ...
- User Thomas Jung on The "One Million Records" Challenge: >I worked with WebDynpro sometime ago. A...
Archives
About This Site
Welcome to the premiere community site for all things RIA sponsored by O'Reilly Media and Adobe Systems Incorporated.
About Us
Meet the Experts
Meet Our Contributors
Send Us Feedback

First let me say that I really enjoyed your guest spot on RIA Weekly. I totally agree that moving RIA technology from the consumer to the enterprise space means that the technology will be faced with a different type of scaling challenge and it is going to be interesting to see what kinds of adaptations will take place.
I enjoy doing these kinds of extreme architectural experiments because it does make you think differently as a programmer and I love giving server administrators heart attacks.
So I was again inspired and wanted to give 10million rows a try. It resulted in a crazy big session state (not surprisingly) but did complete.
http://flickr.com/photos/tjung/2743001410/
I have the unfair advantage of a smallish server running my DB and App. server though. I'm pretty impressed that Curl was able to chew through a 1 million row CSV given the hardware you had it running on.
As to your question about Web Dynpro being RIA, Fat, or Fit client - it is closest to the last one. Web Dynpro produces a metadata based description of the UI and then different rendering engines can be applied to the same application at runtime. The video from my blog is the AJAX/DHTML rendering engine - but we also have an Adobe Flex based rendering engine and a .Net Desktop Client based rendering engine.
I have been playing with the Microsoft Grid for Silverlight and with the support for data and ui vitualization 1m records are displayed with no problem. Actually - you can display 15m records - it doesn't matter because of the virtualization support.
hi
in Adobe Flex, i've created a component that could handle virtually unlimited number of rows and columns. (limited only by biggest int value possible and physical memory)
http://v2.idubee.com/IdubeeClient.html
Press CTRL+m to insert million records
One cell takes only 8 bytes, and render time is O(1) (constant) for any number of cells.
Standard Flex DataGrid component also quite fast.
I worked with WebDynpro sometime ago. As I remember, it used some kind of ActiveX, so it's pretty much obese :)
>I worked with WebDynpro sometime ago. As I remember, it used some kind of ActiveX, so it's pretty much obese :)
Web Dynpro's main UI libraries don't use any ActiveX. The UI elements are implemented in AJAX and DHTML. We do have a couple of specialized controls - for example Microsoft Excel Integration and multiple file uploads - that use either ActiveX or Java Applets. These are pretty specific use cases however. Even the Adobe Interactive Forms elements have been changed from ActiveX to JavaScript based. The future direction for UI elements that can't be accomplished is JavaScript/HTML is to use the Islands technology that allows us to embed Flash/Flex and Silverlight directly into Web Dynpro.
We've had "stress tests" like these for SmartClient for several years. Trees are much harder by the way, try those sometime. Or very large numbers of columns (500 or more) - this requires bidirectional incremental rendering to scale. Or data cubes.
It's really hilarious to me to see plugin vendors make so much noise about performance and then be matched or surpassed by Ajax.
Ajax, Flash, Java and Curl scale to 1M+ rows. Now can we stop talking about naive benchmarks like this, because it would be foolish to load this much data up front in the types of applications these platforms target.
Real-world performance in a WAN environment comes from really intelligent data management, which reduces server round-trips, thereby reducing network traffic and database hits.
If you care about performance, evaluate a platform's data binding architecture and it's facilities for re-using already loaded data, managing caches, and performing client-side operations where possible. If you could benchmark, say, the Cairngorm architecture in Flex against SmartClient's (Ajax-based) databinding architecture, you'd find very large performance differences that actually affect real applications.
Hi Charles,
Thanks for commenting on the blog. I would like to see a demonstration of SmartClient handling 1 million records. Can you post an example? You say that Ajax surpasses the plug-ins - I would like to see an actual demonstration.
All the best,
Richard