Evaluation of Singlehanders by World Sailing: Analysis and Recommendations (Part 1)
 |
| Aerial picture of the d-Zero built by Devoti Sailing in Poland. The boat won many races at the Valencia trials, yet was ranked 4th by World Sailing |
The report follows a tender that was issued on May 25 2018, which called for bids to potentially replace the Laser. Read our article Singlehanded Dinghies for the 2024 Olympics: Selection Criteria and Potential Contenders for more background on the invitation to bid by World Sailing.
World Sailing's evaluation panel is composed of board members, staff, and members of the equipment and events committees as well as of the medical commission. There was also a coach nominated by World Sailing’s coaches commission. In total, there were 12 people involved. In addition, there were 11 sailors that were selected to participate in the sea trials that took place in Valencia, Spain, on March 11-15.
As previously reported in this blog, this exercise was going to be challenging, as numerous considerations are to be taken into account. The reference document for the re-evaluation is the « Equipment Committee Criteria for the re-evaluation of equipment under Regulation 23.6 » Yet, the evaluation panel was given ample latitude to come up with its own criteria and evaluation matrix, which where not known in advance by the bidding companies.
In October 2018, World Sailing announced it had received 8 compliant bids, but did not name them. Following inspection of building facilities, it shortlisted 4 boats: the Laser, the Melges 14, the d-Zero and the RS Aero. This brought World Sailing to organize the sea trials, which took place in Valencia, to compare the boats.
The report issued by the evaluation panel concludes that 2 boats only are considered suitable for the 2024 Olympics: the Laser and the RS Aero. They are respectively given overall scores of 69% and 80%. The d-Zero and the Melges 14 are judged as unsuitable. These overall scores are the result of scores given on 13 criteria, but with different weights, that were assigned by the evaluation panel.
Now, how did the panel reach these overall scores? One needs to get into the details of the report, and this is done in this first part of this article. The presented analysis finds many flaws in the report. There are enough flaws to cast doubt about the validity of the conclusions of the report, which is consequently questionable as a guide for decision-making.
In the second part of the article, recommendations are be made to attempt finding a suitable way forward.
PART 1: EVALUATION OF THE EVALUATION PANEL REPORT
We provide here an analysis of 7 out of the 13 criteria against which the 4 boats were assessed. The analyzed criteria have been chosen because they have the highest weight in the scoring matrix developed by the panel. Together, those seven criteria represent 84.5% of the points in the scoring; the remaining 6 criteria, not analyzed here, represent 15.5%.
The analysis below shows that the scoring is in most instances highly questionable. Yet, this analysis does not offer an alternate scoring, as World Sailing has not made available all the material produced in the context of the Valencia trials - such as the filled out questionnaires, records of commentaries by sailors, commentaries by the staff and all those involved in the process.
As previously indicated, the second part of the article will provide suggestions as how to proceed, given the weakness of the evaluation report as a guide for decision making.
The criteria are presented here by declining order of importance in the scoring matrix developed by the evaluation panel. It's a bit long, and sometimes redundant, but it's a consequence of the report and the overlaps that exist between certain criteria.
Design Evaluation - Athletic Suitability: 15.75% of total scoring
WS Scoring: A (Aero) 4.0; L (Laser) 4.5 ; M (Melges 14) 2.5 ; D (d-Zero) 3.7
« Refers to the evaluation of the athletic suitability of the equipment considering the ergonomics and if success is more related to athletic superiority and tactical understanding than technical knowledge of the specific equipment. »
For this criteria, the Laser won, followed closely by the Aero. For the Laser, « MNA sailors acknowledged their experience and familiarity with the boat. » « Success is more related to athletic superiority and tactical understanding …» And for the Aero, « The light hull requiring sensitivity in boat handling, was considered by MNA sailors as rewarding higher sailing skills and tactical knowledge. »
The report is unfortunately silent on many aspects of athletic suitability. The question of ergonomics is not properly addressed. It is well known for example, and well documented, that there are injury issues with the Laser, particularly in the Radial. The hiking position in the Laser is a well known issue, as the boat, with its flat deck (iron board like …), was not initially intended to be Olympic. Hiking pads are used by about all sailors to alleviate the pain, and avoid reduced blood circulation.
There are some commentary about ergonomics in the Medical Observer Report, which stresses that the Laser’s cockpit is relatively narrow and slightly deeper. The position for downwind sailing is seen as more challenging in the three other boats. « In downwind conditions, the Laser allows for a more comfortable position than the three other boats, where the crouched posture induced by the wide cockpit configuration may facilitate knee strain. » Yet, the report mostly singles out the Melges 14 for this question, and awards the boat a 2.5 score, while the notoriously un-ergonomic Laser receives a 4.5 - the best score of the 4 boats.
What the report fails also to do is to connect the dots. Injuries, particularly in the Radial, are linked to the too high power of the rig/sail for most women. While the report acknowledges that the Radial is not suited for the physique of most female sailors, it still scores the Laser with a 4.5 for athletic suitability.
While the Laser Standard is ok for middle-weight sailors (of approx 82 kg, with unfortunately a very narrow weight range), the Laser Radial is in fact NOT athletically suitable for most female sailors.
It must be noted that on boats like the d-zero and the Melges 14, the width is intended to increase the righting moment, and therefore the speed of the boat. It’s also supposed to widen the sailor weight range. There are obviously trade offs involved, but the real performance, the real speed of the boats was not assessed in the context of the trials (see below), as speed was not even taken into account in the evaluation of the performance of the boats.
Another aspect related to ergonomics that was not analyzed in the report is the risk of concussions. The danger of concussions, even mild ones, is increasingly recognized in sports and in the academic literature. There are now youth boats where helmets are compulsory at least in some countries. This includes the O’Pen Skiff (ex O’Pen Bic) and the Nacra 15.
The risk of concussion is related to the height of the boom. The 4 boats should have been evaluated for that risk. Also, the risk may differ according to the boom material - the Laser has a aluminum boom while the three other boats have ligher carbon / composite booms. Note that wearing a helmet in some boats is very challenging, because of the low boom. Again, this is something that the evaluation should have assessed, to favor a boat where wearing a helmet is feasible.
Verdict: Highly Questionable Scoring
Cost Evaluation - Retail Price - 13.75%
WS Scoring: A 3.6; L 4.5 ; M 2.5 ; D 5
« This subgroup seeks to evaluate the price of the equipment taking into account not only the retail price of the boat but considering the price of replacements to address the life-cycle costs. See Appendix 4. »
The comparison made in the report is based not only on the retail price of the boat (60% weight) but also on the price of the spares (40%). Yet the price of the spares assumes one spare of each type (boom, mast, sail, daggerboard, rudder, etc.). So there was no attempt to estimate the real cost to sailors. The report claims to evaluate the real life-cycle costs, but actually fails to do so.
In terms of boat acquisition, the d-Zero is the cheapest of the 4 boats, despite the fact it uses composite materials for the hull, has carbon spars from a respected manufacturer, and has a laminate sail, with longer competitive life than dacron sails. Yet the d-Zero does not get a clear advantage from this, as while scored at 5, the Laser gets a 4.5 score and the Aero a 3.6 score.
Regarding the Laser, although the report mentions the low longevity of the Laser sails, the cost associated with the multiple sails that must be bought by top sailors (easily 10 or more sails per season) is not factored in the analysis, and it is known that the price of the Laser sails is excessive for what they are. This helps the Laser get a high score of 4.5, while the reality is that Laser sailors spend unnecessary large amount of money on the sails, that typically are only competitive for one or two regattas.
And regarding the Laser hulls, the report indirectly admits there is an issue with the longevity of the hulls, but this is not factored in the cost analysis either. Instead, the report comes with the astounding assertion, on page 5: « The hull is durable beyond its competitive life making cheap boats available to many sailors. » This is an admission not only that the boats don’t last long, but also that they loose substantial value after their competitive life, as they are then « cheap » to buy by other sailors.
To be noted is that, from a sailor’s perspective, the price differences are not at all marginal. For example, the price of the d-Zero, with the large rig, is listed in the report at GBP 4448.3 while the RS Aero 9 is listed at GBP 6040.6 and the PSA Laser Standard at GBP 5515.4. This is over US$2,000 cheaper for the d-Zero compared to the RS Aero in terms of acquisition cost. And the running costs, for competitive sailors, may be lower for the d-Zero, in part thanks to the laminate sails.
The comparison also takes into account the pricing of the Laser by LaserPerformance, but by the time the report was released, LaserPerformance had already been decertified by the Laser class. The now dominant Australian Laser builder has significant higher prices for its boats, and as the bulk of the demand for new Lasers is in Europe, there are additional costs in terms of transportation. The higher pricing of the Australian boats is indicated in the report, which was not updated to reflect the new situation regarding Laser builders.
Interestingly, on page 11 of the report, the « MNA Sailors’ Impressions » about the « value for money » are indicated, and the Laser wins, even against the D-Zero, which features, at a lower price, advanced building materials, a laminate sail and carbon spars. The Laser gets from these MNA sailors a score of 4.1 while the D-Zero gets only 3.6 - which suggests that at least some of these sailors may favor the Laser, irrespective of the merits of the other boats.
Verdict: Highly Questionable Scoring
Quality Evaluation - Standardization of Equipment: 13.50%
WS Scoring: A 5 ; L 3 ; M 3 ; D 3
« For this section standardization refers to the evaluation of the standardization among produced equipment, accounting for existing quality controls worldwide, measurement procedures at events, building specifications, and the evaluation of Class Rules towards achieving equal equipment including the control of fittings permitted. »
This criteria is obviously very important for an Olympic class to be sailed globally. But it’s also an extremely demanding criteria to assess. First, there was no testing done on large samples of equipment in the context of the report. All the new designs are recent - about 5 years old or less - and their actual durability is not yet known, especially in the context of a usage by top Olympic level sailors.
In addition, while there is pressure by World Sailing, to address competition concerns, to have multiple builders and enhanced competition among them, two boats have only one builder (Aero and d-Zero) and two have just two (M14 and Laser). In the case of the Laser, the boats that were present in Valencia were actually from LaserPerformance, i.e. the builder that had already been decertified by the time the report was published.
It’s worth commenting here on the Laser. There is a well documented history of inconsistent building of the boat. This is recognized in the report, which states that « although compliant with their construction manuals and quality controls, the presented tolerances were considered by the Evaluation Panel as too high. » And to continue: « The supply of equipment for Olympic events and other major events mitigates the poor standardization, however tighter tolerances and higher controls are deemed required. »
In addition, recent developments have brought to the fore that Lasers may have been voluntarily built at a higher specification in Australia. A « Defect Notice » by the laser class ILCA states that 2,280 boats produced between 2006 and 2015 are « known not to comply with the manufacturing specifications of the Laser Construction Manual. » « The defect is an additional layer of approximately 300g/m2 chopped strand mat (« CSM ») included in the hull laminate forward of the centerboard case to the bow of the boats …. the existence of which has been confirmed by visual inspection and technical analysis of a deconstructed sample PSA boat. » This information was not taken into account in the WS report, although it was public prior to its release.
What are the implications? It seems much easier to implement consistency in production when only one builder is involved, which is the case for the Aero and the d-Zero. When several builders are involved, there was no attempt by the evaluation team to assess the consistency between the various builders. In the case of the Laser, now that LP has been decertified, it would mean comparing the boats produced in Australia and Japan. In the case of the M14, it would mean comparing the boats produced in the US and Portugal.
As for key components, particularly the spars and the blades, it would be justified for relatively large samples (say 10 or 20 units) to be tested, ideally on a random basis, to assess the consistency in production. Without such testing and sampling, it’s not possible to confirm there is actual consistency in production and to come to a conclusion.
Instead of adopting a big picture approach to this critical issue of consistency, and demonstrate some humility given the huge task to assess it, the report points out a minor problem with one mast on the M14. The evaluators blame the production process, but did not actually verify it in Portugal.
Regarding the d-zero, despite the negatives found in many parts of the report, the section notes « The samples of the D-Zero seen at the trials were nicely finished and inspections revealed that there were only very minor differences between presented parts considered acceptable by the Evaluation Panel. »
The goal of Devoti of having multiple builders is being challenged in the report. But this is the same kind of problem that the Laser has been facing for 50 years.
For RS Sailing, there is only one building facility, yet World Sailing is confident that standardization would be achieved if there would be additional facilities. But how to be certain? Yet, Aero gets a high score.
In sum, the scoring regarding standardization seems pretty speculative on the part of the evaluators.
Verdict: Highly Questionable Scoring
Design Evaluation - Performance - 11.25%
WS Scoring: A 3.9 ; L 4 ; M 2.3 ; D 3.0
« Refers to the evaluation of the performance of the equipment considering all equipment items including the design of the hull, appendages, sail and running rigging. It includes considerations regarding maneuverability, responsiveness of the boat and the effectiveness of the outfitting of the systems. Suitability of the presented equipment sizes is also considered in this section. »
« For this subgroup, the presented result of the evaluation was directly obtained as the outcome of the MNA sailors score on the subject. »
The comparative performance of the various boats is clearly what what people have had in mind for months. How fast are those boats. Are they really that faster than the Laser? Were the Valencia trials going to provide us an answer? Of course not!
The way World Sailing has defined performance in the context of this evaluation is such that even an Optimist could beat a SailGP F50!
The details are unknown, but the above quote aspects such as maneuverability, responsiveness of the boat and the effectiveness of the outfitting of the systems. Nothing to do with how most do perceive performance.
The result of the approach to performance chosen by World Sailing is that … the Laser wins! Yes, after nearly 50 years, not a single one of its challengers can beat the Laser! Extensive research and development, using carbon spars, mylar sails, lighter composite hulls is useless! It does not even make those boats more performant than the Laser! This does not make any sense of course.
The report does not even include a single criteria geared towards the actual speed, the actual performance of the boats!
The question of optimum sailor weight is however covered in this criteria. For the Aero, there report states that « the presented sail areas were considered better suited for higher crew ranges, » but the report does not highlight the fact that, according to its own estimates, the RS Aero 7 would be even less suited than the Radial for typical female sailors.
 |
| Ideal Sailor Weight Ranges Estimates for the 4 Boats by World Sailing Evaluation Panel From left: Aero, Laser, M14 & d-Zero |
For the Laser, the report mentions that « the women’s rig promotes a larger weight and height than desirable for worldwide average women. » The report also states that « The rig and sails are considered the downside part of the boat and of poor quality for the associated price. » Known flaws of the Laser hull, including its too high weight and the inefficient water evacuation system, are not even mentioned.
The Laser wins for performance!
(bad tongues will say that one needs to thank LaserPerformance for that as they provided the boats :)
For the Melges, which is just one inch longer than the Laser, the report states that « the hull size remains too large » … « making the boat unsuitable for smaller rig sizes. »
For the d-Zero, the presented rigs were also considered too powerful and the boat was seen as better suited for inland sailing in flat waters, because a narrow, low volume, bow.
What to make of all of this? By its own admission, the report indicates that all the four boats were too powerful for typical female sailors, with the rigs presented in Valencia, including the Radial and the RS Aero 7, which obtain the highest scores.
This can be seen from the videos and pictures made available by Farevela, with the sailors in full hiking in 10 - 12 knots, while the boats are supposed to also sail in 25 knots. Of course, to properly assess the optimum sailor weights, much more time was needed, and only high level athletes should have been involved in the sea trials.
For female athletes, the actual challenge is to move down the optimum weight range, as all the tested rigs, including that of the winning Laser, simply are too powerful for most women.
For male sailors, there are estimated weight ranges too, but the only reliable one is for the Laser, which was known prior to the trials. Their situation actually differs greatly from female sailors, as moving a bit up the optimum sailor weight range for men is not much of a problem and may actually be desirable, as it could allow sailors of say 85 to 90 kg to compete at the highest level. This would allow to accommodate at least some of the sailors who cannot contemplate a career in the Finn any longer.
And what about actual performance, i.e. speed? The last day of the Valencia trials featured races among the three contenders - with the Laser staying ashore (no reason provided …) Laser Standard world champion Pavlos Kontides reportedly won all the races, irrespectively of the boat he was using (Aero, Melges, d-Zero) while all the d-Zeros reportedly won all the races with female athletes, irrespective of which sailor was sailing it.
You may also want to refer to the Portsmouth index data, showing that the Laser is much slower than the Aero 9 and the d-Zero (approx. 4 minutes per hour of racing)
Also to be noted is that the question of righting boats after capsize is not analyzed in the report. This is of importance, particularly for youth sailing, and the boats should have been tested in that regard as they are reported to behave very differently.
Verdict: Highly Questionable Scoring
Design Universality: 11.25% of the total scoring
WS Scoring: A 4; L 3 ; M 0 ; D 1
« Refers to the evaluation of the universality considerations related to the design of the boat. These include the evaluation of the simplicity of the systems, the pathway classes and the suitability to serve as youth equipment. It includes the evaluation of sailor’s weight and height range, ease of launch, retrieval and towing and ease of assembly. »
This is another criteria with questionable scoring. Melges scores zero, with as argument that the boat is « suited for larger persons » and « the hull size was deemed unsuitable. » The length of the M14 just one inch longer than that of the Laser. It is just 1% longer! The boat is a bit wider too, to achieve a better righting moment and make it faster, but what makes the boat look big is the design of the cockpit, more than its dimensions. Yet the boat gets a zero score from the panel! Zero!
And for the same criteria, the d-Zero gets a score of 1. Why? « With a mainsheet traveler and options to adjust the mast rake the boat was considered the option with the most complicated arrangements and with the potential for competition for technical knowledge. » states the report.
This point illustrates big time how weak this report is. First mast rake, even in a supposed strict one design boat like the Laser, plays a critical role, to the point that the Australian builder sells boats with various mast rakes, and has a guide, available online, to help select, at the time of purchase, the hulls according to the desired mast rakes.
Any competent Optimist sailor knows about mast rake. It plays a critical role in the Optimist. Yet the report fears a « potential for competition for technical knowledge. » So what any competent Optimist sailor can do is considered too complex for single-hander Olympians who may devote 10 years of their life to learn the specifics of their boat!
« The testing and research on the use of the different settings would result in a disadvantage for teams with less resources, » claims the report.
The report recognizes that an adjustable mast rake can serve weight range. This was an explicit consideration in the bidding documents put forward by World Sailing. But the report prefers to consider that it is too complicated, as it « would increase the boat specific tuning knowledge required to be competitive. »
The report also gives preference to the Dacron sails versus the Mylar sails, hence prefers the Laser and the Aero in that regard. But it recognizes that the competitive life of mylar sails is longer. Again, one wonders what kind of boat is being selected: a boat for the Olympics or a boat for your local regattas?
And then there is the supposed risks associated with the chocks used on the d-Zero to adjust the mast rake. « These small pieces could be lost during transport or on the water and may be difficult to replace in some areas of the world increasing cost. » Reality is that any piece of plywood would do the trick. There are obviously no such risks whatsoever, but still the report highlights this as a negative for the d-Zero.
Mast rake adjustment is actually a very positive feature of the d-Zero, because a) having a consistent mast rake out of any factory is extremely difficult to achieve, especially for Olympic level sailings where millimeters count, and b) adjustable mast rake enables to accommodate a wider sets of sailor weights and physiques.
There is nothing complex with mast rake adjustments, but the report dismisses this feature, and by doing so, like with its dismissal of the mylar sails, unduly favors the Aero and the Laser.
What has not been done by World Sailing is to go to measure mast rakes in the various building facilities, on a large sample of boats, and figure out the level of consistency. And the mast rakes measured at the Valencia trials were not published.
As previously indicated, it is known that the Lasers produced by the now dominant builder - Performance Sailcraft Australia - have significant variability in their mast rakes and can actually be bought with a specification of the mast rake, defying thereby the very concept of strict one design sailboat. What’s the degree of consistency with the Aero and the Melges 14 remains unknown.
At least, with a system like the one on the d-Zero, there is no such problem. It makes things much more transparent and fair.
As for existence of a traveller, this is obviously a standard feature in many many boats, including youth sailing boats, yet it’s seen as a negative of the d-Zero. Too complicated claims the report!
Verdict: Highly Questionable Scoring
Cost Evaluation - Universality Considerations - 10%
WS Scoring: A 4 ; L 3 ; M 3 ; D 1
« Considers access to the equipment, distribution networks, availability of charter equipment, mid & long-term distribution scenarios and durability. »
This is maybe one of the least contentious scoring, although it’s hard to believe that Devoti would not adopt an adequate distribution strategy if the d-Zero would become Olympic.
Note also, while the report was released after the decertification of LaserPerformance, it does not take into account the trouble in which the Laser is presently, with shortages of new boats, and increased prices, resulting from the decertification by its class ILCA of the largest builder.
Verdict: Questionable Scoring
Quality Evaluation - Quality of Production 9%
WS Scoring: A 4 ; L 3 ; M 3 ; D 1
« Quality is a critical characteristic in all Olympic equipment and together with standardization should be highly valued for universal equipment that must serve all the nations in our sport equally. The quality of the boats was investigated in depth before and during the trials. This subgroup refers to the evaluation of the manufacturing methods, the consistency of equipment in terms of weight and finishing of the product, the materials, tolerances, durability, and stage of development (proven vs experimental). »
Let’s start here with the scoring for the Laser, which features one of the most non-sensical assertions of the report, already mentioned previously.
« The hull has proven to be durable beyond its competitive life making cheap boats available to many sailors »
The report essentially admits, without specifying the actual competitive life of Laser hulls, which are known to last one or two seasons at the top level of competition, that the hulls are actually « durable » beyond those one or two seasons.
And then, good news for everyone, this makes « cheap boats available to many sailors. »! Instead of blaming the Laser and its 50 years of existence for not having managed to produce boats with a long competitive life, the report spins it as a positive. Cheap boats are made available to many sailors! And this brings a score of 3 to the Laser, nearly as high as that for the Aero, and equal to the Melges!
All the issues of standardization with the Laser, the fact that two types of Lasers were apparently produced for years, the fact that the current building facilities in Australia and Japan were not inspected in the context of this report, and that the boats that were checked originated from the builder that was shut down by the class, are not even mentioned in this section of the report! Instead, the Laser is offered a score of 3 - thanks to its ability to provide cheap second-hand hulls!
For the d-Zero, how is the panel justifying the very low score of 1. While the report states that Devoti « has a proven record of quality products that have proven durable, » it indicates to some insufficient quality controls. An issue is also mentioned regarding the raised deck support for the upper rudder fitting. Does that warrant a quality score of 1, in light of the score of 3 awarded to the Laser? …
The score of 3 on the Melges seems pretty harsh too. The Melges 14 seems penalized for a mast issue, that seems minor at worst. As indicated previously, the issue of consistency in mast production, including bend, was not properly analyzed by the evaluation panel, as no large samples of spars have been tested for any of the 4 contenders.
Verdict: Highly Questionable Scoring
The 6 other criteria, which have much lower weights in the overall scoring, are not analyzed here, but they probably also raise many issues.
In the second part of the article, we will attempt some suggestions about what to do next.
