Replacing Qt XML validator by custom validator

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Replacing Qt XML validator by custom validator

Leon Vinken
In the past we have had several complaints about loading large MusicXML files taking too long and not providing any any feedback. This is caused by the Qt XML validator used: it is not very quick and it does not enable any feedback.

To get a feel for possible improvements I have built a prototype custom MusicXML validator, with very encouraging results: it validates about four times as quick as the Qt version, with the same result (pass or fail) on the almost 1000 files I have run through it.

Some numbers: the largest sample file I have contains about 45.500 notes. It takes 74.3 seconds to validate, while the actual MusicXML import takes only 6.7 seconds, resulting in 81 seconds import time. The prototype custom validator runs in 18.4 seconds, reducing the import time to 25 seconds.

Given these results, I would like to propose to finalize the custom validator and replace the Qt validator by it. This will effectively make MusicXML import three times quicker and would allow providing feedback while importing in future. Note that this will still require a considerable effort, as the current version is not yet "production quality".

Opinions anyone ? Would this be worthwhile ?

Regards, Leon.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Replacing Qt XML validator by custom validator

lasconic
Administrator
Hi Leon,

Did you wrote the validator yourself? If no, what's the basis or the lib you used?

lasconic

2016-01-09 22:05 GMT+01:00 Leon Vinken <[hidden email]>:
In the past we have had several complaints about loading large MusicXML files
taking too long and not providing any any feedback. This is caused by the Qt
XML validator used: it is not very quick and it does not enable any
feedback.

To get a feel for possible improvements I have built a prototype custom
MusicXML validator, with very encouraging results: it validates about four
times as quick as the Qt version, with the same result (pass or fail) on the
almost 1000 files I have run through it.

Some numbers: the largest sample file I have contains about 45.500 notes. It
takes 74.3 seconds to validate, while the actual MusicXML import takes only
6.7 seconds, resulting in 81 seconds import time. The prototype custom
validator runs in 18.4 seconds, reducing the import time to 25 seconds.

Given these results, I would like to propose to finalize the custom
validator and replace the Qt validator by it. This will effectively make
MusicXML import three times quicker and would allow providing feedback while
importing in future. Note that this will still require a considerable
effort, as the current version is not yet "production quality".

Opinions anyone ? Would this be worthwhile ?

Regards, Leon.




--
View this message in context: http://dev-list.musescore.org/Replacing-Qt-XML-validator-by-custom-validator-tp7579577.html
Sent from the MuseScore Developer mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
Mscore-developer mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mscore-developer


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
Mscore-developer mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mscore-developer
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Replacing Qt XML validator by custom validator

Leon Vinken
Yes, I wrote it myself from scratch. Currently about 4500 lines of code. It understands just enough of the W3C XML schema language to correctly validate MusicXML.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Replacing Qt XML validator by custom validator

Mark Stanton
In reply to this post by Leon Vinken
On Saturday 09 January 2016 14:05:41 Leon Vinken wrote:
> In the past we have had several complaints about loading large MusicXML
> files taking too long and not providing any any feedback.

I'd vote for it.

And, to avoid the problems associated with having custom built stuff of our
own, how about contributing it to the Qt project? :)

Mark

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
Mscore-developer mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mscore-developer
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Replacing Qt XML validator by custom validator

lasconic
Administrator
Hi,

Honestly, I didn't expect this answer. 
Writing an XML validator from scratch sounds like a bad idea when there are several already existing (Qt, Xerces c++, libxml2 etc...) with already many man-days invested, hundreds of projects using them and years of bug fixes. May I ask why you chose to rewrite one instead of optimizing an existing one?

lasconic

2016-01-10 11:02 GMT+01:00 Mark Stanton <[hidden email]>:
On Saturday 09 January 2016 14:05:41 Leon Vinken wrote:
> In the past we have had several complaints about loading large MusicXML
> files taking too long and not providing any any feedback.

I'd vote for it.

And, to avoid the problems associated with having custom built stuff of our
own, how about contributing it to the Qt project? :)

Mark

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
Mscore-developer mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mscore-developer


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
Mscore-developer mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mscore-developer
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Replacing Qt XML validator by custom validator

Leon Vinken
A valid question. When I started I had no intention to re-invent the wheel, but first spent considerable time investigating the existing solutions. I could not find one that met my criteria: easy to understand and use, fast and easily able to provide feedback while validating.

While building a prototype to investigate what would be required to solve the issue and what would be the expected performance, I found that while the W3C XML schema language is quite complicated, the part of it required to correctly handle MusicXML is limited. So I built little bits from time to time, and as a result, I now have a (simple) prototype MusicXML validator that works, is fast and correctly validates all samples I've run through it.

And to answer Mark's question: it does not make sense to integrate it into Qt, as it serve a different purpose: it is not a generic XSD validator, but one that only works for MusicXML.

Regards, Leon.
Loading...