Evaluation of Mirial 7.0
Mirial Softphone 7.0
Introduction
This report is one of an ongoing series of evaluations of desktop videoconferencing software that is being produced by the Video Technology Advisory Service (VTAS). The tests and evaluation have been conducted in accordance with the principles and procedures set out in the Desktop Evaluations Overview.
Mirial Softphone is a fully featured, standards compliant, desktop videoconferencing client that is capable of High Definition (HD) resolution. It offers many of the features of a stand-alone videoconferencing appliance, but on a Microsoft® Windows® personal computer (PC) or Apple® Mac® computer. Testing of this product was carried out on the PC only. It was found to interoperate well with a number of different manufacturers’ equipment (including the market leaders: TANDBERG, Polycom®, and LifeSize®) and it offered such good quality that it was not always apparent to the person at the other end that this was a webcam and speakers setup.
Basic Details
Application: Mirial Softphone
Software Version Tested: 7.0.2 -7.0.7
Product Website: http://www.mirial.com/products/Mirial_Softphone_HD.html
Test Dates: July - October 2009
Testers: Geoff Constable and Roger Gachago, with help from Chris Bodley and Shafiq Latif
Mirial Softphone is an application that can be installed on a PC, laptop or Mac, and allows users to make video and audio calls between other Mirial Softphone users and any other PC clients, or videoconferencing equipment, that support the H.323 or SIP standards. This makes it extremely versatile and fully compatible with JANET(UK)'s supported services. The client is easy to use, and offers many of the features found in dedicated videoconferencing equipment, including:
- H.239 data-sharing (sending a presentation, or desktop application, in addition to the camera image)
- higher than standard resolutions (where the PC is powerful enough to support this)
- sending a desktop application as an alternative to the camera image
- hosting two calls simultaneously, allowing a three-way videoconference
- recording calls and saving them as a windows media format file for playback or distribution
During the testing period, a number of small upgrades were released and installed; however, these did not change the overall functionality and performance significantly.
Costs
The software is paid for by a simple purchase, with one single version offering all features. As with most videoconferencing equipment, it is retailed by third-party resellers. The purchase process is simple, involving ordering the product from a website, then receiving an e-mail with a link to a download and a licence to use the product. A free (fully featured) thirty day evaluation copy is available for download from the product web site.
Standards and Security
Mirial Softphone supports a number of videoconferencing standards including H.323 version 4, H.239 (data-sharing), H.281 and H.224 (far end camera control). The software is also fully compliant with Internet Engineering Task Force (IETF) Session Initiation Protocol (SIP) standards. The audio standards supported include the most basic (G.711) through to the high quality G.722.1 Annex-C. Similarly, with video the required H.261 basic encoding is supported, as are H.263, H.263+ and H.264. H.264 is supported at the following resolutions: Common Intermediate Format (CIF) (352 × 288 pixels); 4CIF (704 × 576 pixels); 720p (1280 x 720 pixels); and 1080p (1920 x 1080 pixels). It should be noted that the use of these formats is dependent on the bandwidth available on the network, and the particular PC hardware on which Mirial Softphone is installed. Encrypted calls are possible using the SIP client, but Mirial Softphone does not encrypt calls when using H.323.
The issues that H.323 has in overcoming Network Address Translation (NAT) and H.323 are present when using that standard to communicate across a firewall and/or NAT boundary, but it is possible to limit the ports used by the software, and also to configure NAT awareness (see the VTAS document NAT, Firewalls and Videoconferencing, http://www.ja.net/documents/services/video/vtas/h323_border_traversal.pdf for a full explanation of these issues). Neither of these options was tested during the evaluation for this report. For the technically minded, it is also possible to configure a Differentiated Services Code Point (DSCP) for network quality of service support, but again, this was not tested during this evaluation.
Installation
The programme is supplied across the Internet. Once the purchase procedure is complete, an e-mail is received that gives the address from where the software can be downloaded. This is a simple process, and the installation file (version 7.0.3) was 10.4Mb (the equivalent of two or three MP3 music files). The purchaser also receives a zipped licence code that must be extracted to a folder on their computer. During the installation process, it is necessary to browse the computer to the location of the licence file, and this completes the full software installation. One of the Microsoft® Windows Vista® test PCs froze during this stage of the installation process, but a simple restart fixed the problem and the software was then ready to use. In all, the installation process takes a couple of minutes.
On first use, the user is prompted to enter various configuration details. It is possible to accept the defaults for these, although it is advisable to change the network default from 384/Local Area Network (LAN) to match the network speed available at your location. Although these configuration details are presented to the user during the first use of the software, it is possible to access them at any time by selecting the configuration icon at the bottom of the main window. If the user wishes to attempt to use the software in High Definition resolutions, it is necessary to enable 4CIF, 720p and 1020p. When configuring the software to use an H.323 gatekeeper, it is necessary to have the details of the IP address of the gatekeeper and the E.164 number to hand (the E.164 number is entered under Phone Number). It is suggested that the best approach here is to familiarise oneself with the software while it is on the default settings and then attempt to extend the bandwidth and video formats attempted.
Figure 1: Mirial Softphone Configuration initial options screen. This is displayed automatically on first use of the software, but available at any time by selecting the Configuration button.
Documentation
Two guides are supplied with the software – a Quick Start Guide and a full user manual. The Quick Start Guide is an illustration of the user interface with brief explanations of the functionality of all the various buttons. The Quick Start Guide does not explain all the configuration options.
The full user manual is very thorough and covers all the functionality of the software and all the configuration options available. In addition, there is a series of Frequently Asked Questions (FAQs) on the product web site, covering potential installation and usage problems.
On first use, it was necessary to adjust the Windows® firewall settings to allow the program through the firewall.
Figure 2: On first use, the Windows Firewall (or equivalent) may block the programme. Correct this by clicking Unblock
User Interface
The Mirial Softphone user interface is easy to use once mastered, although not particularly intuitive at first. The icons, buttons and writing on the user interface were very small for this reviewer, and it was useful to print out the Quick Start Guide and have it to hand in order to become comfortable with the various buttons available. Basic functionality is easy enough, but the dialling area is very small, as are the telephone icons that turn from red to green when you are in a call.
Tooltips (small pop-up windows with descriptive text, which appear when the mouse is hovered over an icon or button) are available, which are very handy given the size of some of the buttons and icons on the user interface.
The main user interface, which displays the local view when not in a call, is taken up mostly by the incoming picture, but the top right hand corner has an attached area that allows for call dialling. It would make the software more usable if this could be hidden while in a call and accessed when required. Instead it is present at all times, which feels unnecessary while a call is in progress.
The IP addresses and numbers dialled are remembered and offered as auto-completion in a drop-down list attached to the dialling field. Calls are also logged and details are available in a pane that drops down below the main interface.
The trappings of the user interface disappear when the picture is in full screen mode, and all use and testing for this evaluation was done with the software in full screen mode whenever possible. To return to the user interface from full screen mode, simply press Escape (Esc).
Although the user interface is quite easy to use, testers did have some minor issues with it:
- the lists of contacts, previous calls, running applications for sharing, and recordings are all shown (when activated) attached to the bottom of the application window. It would be more useful if they could be detached as an independent window
- it would be useful if the dial pad area of the application window could be hidden when not in use, or presented to the user as a separate window that could be minimised
- the user interface is not configurable – colours, font size, etc. are all fixed
- the font size and the buttons for various actions are too small
- tooltips seemed to work on some installations and not others
Figure 3: The Mirial Softphone interface when not in a video call, displaying the local view and dial pad area
Figure 4: The Mirial Softphone interface when in a video call. The picture can be resized from the bottom right hand corner, and the self-view picture can be dragged to anywhere on the desktop (and resized if required)
Figure 5. The Mirial Softphone interface while in full-screen mode
Connectivity
As mentioned above, it was necessary to change the settings on the firewall of the local PC. It was also necessary to change the settings on the organisational firewall to allow inbound and outbound calling without an H.323 gatekeeper. In a NAT environment this would not work without further adjustments to the firewall. However, there was no problem in registering with the local H.323 gatekeeper and then dialling out (and being dialled) using the JANET Videoconferencing Service (JVCS) E.164 dialling scheme.
The software interoperated with the JVCS without any problem, and was able to pass a JVCS Quality Assurance (QA) test. It would be possible to register a Mirial Softphone as a fully registered venue or desktop venue on the JVCS. A number of successful test calls were made involving the JVCS.
When not registered to the gatekeeper, and using a public, routable IP address, it was possible to send and receive calls using IP addresses once the necessary changes had been made to the firewall. Successful calls were made at all bandwidths, and to a number of different manufacturers' equipment. See Appendix 2 for a list of all of the CODECs that the Mirial Softphone called successfully during these tests.
It was noted that calls connected much faster at the higher bandwidth levels.
Call Procedures
It is possible to make calls by both IP address and E.164 number. Calls will be made at the default network speeds set up in the configuration process, but it is possible to configure bandwidths on a per-call basis by right-clicking on the telephone icon after entering the E.164 number or IP address and before dialling.
The software allows you to choose whether or not to use an H.323 gatekeeper. Depending on their configuration, some gatekeepers do not allow calls to be made by IP address when the endpoint is registered to the gatekeeper. If this is the case, E.164 numbers will have to be used for dialling, or the software will need to be de-registered for the IP address to be dialled.
The Contacts List
It is possible to redial a site that has been dialled previously by double-clicking the entry in the Call log, which lists previous incoming and outgoing calls. Similarly it is possible to have a list of contacts and to place a call by double-clicking on a contact in the contacts list.
It does not seem possible to automate the creation of the contacts list; for example, there is no prompt to add someone to the contacts list following a call and there does not appear to be a means to import contacts (from Microsoft® Outlook®, or a networking site, for example). It is also not possible simply to add someone to the contacts list from the call log list. So it is a little tedious to create an entry for all contacts manually, although there is probably more point in doing so if using the software in a SIP environment, as there is presence awareness (if the feature is enabled and supported locally) and this turns entries in the contacts list green and changes their status to online if they are at their computer and running Mirial Softphone or other SIP client.
Hosting a three-way call
This facility was tested and found to be easy and effective to use. While in a call with a party on Line A it is possible to call or receive a call on Line B. Then, by clicking the conference button, it is possible to host a three-way call. This was tested at LAN speeds and found to be of good quality for all three participants. Each participant has the first called party large in their screen and both themselves and the second party as smaller picture-in-picture displays.
Out of interest, one of the participants in a hosted three-way call dialled out to fourth party and initiated a four-way call. This resulted in a 'daisy chain', in which each participant saw the people they were directly connected to but the party at either end of the chain did not see the person at the far end. However, all participants could be heard, resulting in an effective four-way audio call. Figure 6 (below) shows how this call looked to one participant.
Hosting a three-way call puts additional stress on the network connections between the PCs involved, and on their CPUs. In testing it was found that a two-way 4CIF (high resolution) call dropped to a CIF (standard resolution) call when a third site was brought into the call. When the additional site left the conference, the two-way call returned to 4CIF. So hosting a meeting may mean a lower quality video image is used.
Similarly, data sharing (see below) can also add to network congestion and CPU load. In tests, opening an H.239 shared presentation during a three-way call caused the image quality transmitted from one site to drop even further (to Quarter CIF (QCIF)) while the data was being shared. The quality of the shared desktop, however, was very good and easily legible.
Figure 6: A four-way call
Data Sharing
Mirial Softphone comes with the capacity to share applications (show presentations or documents for example) in two different ways, video sharing or using H.239. Opening and sharing other applications on the PC at either end of the call can affect the overall call quality and particularly the video format in use. This is true whichever method of data sharing is being used.
Video Sharing
If the Presentation button is clicked, a drop-down window appears at the bottom of the Mirial Softphone window, which lists the applications running on the PC. It is then possible to select one of these applications from the list to send to the other end as an alternative to the camera. In tests, the image received at the other end was somewhat blocky and difficult to read.
H.239
This is the standards-based method for sharing documents between two H.323 endpoints. The H.239 protocol for desktop and application sharing is part of the H.323 standard. Use of H.239 allows the endpoint to send a video stream (typically a camera image) and simultaneously send a data image (typically a presentation, but can be any program running on the desktop, or the whole desktop). The H.239 protocol is not enabled by default, and it is necessary to go into the configuration interface and select the CODECs tab before selecting H.239. In all tests, this method of sharing applications or presentations was found to give much better results at the far end. The presentations and web pages shared had far better resolution using H.239 rather than video sharing.
It is only possible to use H.239 between two endpoints if they both support the protocol. If they do, then, providing they have two screens, they will see the presentation on one screen and the presenter on the other. If they have one screen, the presentation will usually take up most of the screen with the presenter in a smaller 'picture-in-picture'. When an application has been selected for sharing, that presentation is always on top of other windows on the screen. Mirial Softphone handles incoming H.239 by opening a separate window for the data stream. This can be moved, resized etc. to the layout that is desired.
Figure 7: Sending a web browser using H.239 data-sharing
Figure 8: Receiving an image from a document camera using H.239
Figure 9: Receiving an image from a document camera using H.239, at full screen on the local PC
Recording Calls
A feature offered by Mirial Softphone, but not by any of its rivals (apart from the free open source H.323 videoconferencing software Macintosh® client XMeeting,), is that of recording calls and storing the recording for playback and review through Mirial Softphone. Alternatively, the recording can be exported in Microsoft® Windows Media Format® (.wmf), which allows the user to review the recording using Microsoft® Windows Media Player®. Exporting the recording to Windows Media Format also has the advantage that the recording can be e-mailed, stored on a Compact Disc (CD) or memory stick, or embedded in a web site. It should be noted that when a recording of a conference including an H.239 presentation is reviewed using Mirial Softphone itself, both the H.239 presentation and remote speaker are visible; but when the same recording is exported to .wmf, the data-sharing element is lost and the shared application is not seen when the file is played back in Windows Media Player.
The recording feature was tested and found to work very well. During a recording, a notification is displayed at both sites that serves as a continuous reminder (see Figure 10, below). This seems a bit unnecessary but there is no way of removing the warning.
Figure 10: A playback of a videoconference in a media player
Figure 11: Playing a recorded videoconference using Mirial Softphone
Quality Testing
These tests were performed on the equipment used for all VTAS desktop videoconferencing testing carried out during 2009, i.e. on the PCs described in Appendix 1.
Audio Tests
All testing was performed in accordance with the outlines described in the Desktop Evaluations Overview document (REF). The standard equipment (i.e. headsets with microphones) was used for all but one of the audio and video tests.
Tests were conducted at 256 kilobits per second (kbps), 512kbps and uncapped bandwidth. These are roughly similar to a poor ADSL connection, a good ADSL connection and an office LAN respectively. Tests were repeated with video muted and video enabled.
Audio levels were found to be adequate in all tests. Audio quality was acceptable in all tests, apart from those at 256kbps when an extra service like data-sharing or recording was being used. In calls using these extra services at low bandwidth on the Windows XP PC, the audio did start to break up at times. This was not apparent at uncapped bandwidth. In general, audio interoperability was good with other H.323 systems, although the Mirial Softphone software only ever negotiated the poorer quality G.711 audio encoding with TANDBERG endpoints. With all other endpoints tested, Mirial Softphone negotiated the better quality G.722 encoding.
Tests were also made using a USB speaker/microphone combination unit, which proved very useful for making the PC suitable for a 'huddle' conference of two or three people.
Video Tests
All tests were made with the picture at full screen. For unlimited bandwidth calls, the bandwidth was set at 1920kbps, for the others a router was set up to restrict bandwidth to the required level, and the Custom settings in Mirial Softphone were set at either 256kbps or 512kbps. All video tests were conducted using the default video format, CIF. In some calls, the video format would fluctuate during the call according to the state of the PC’s CPU – particularly if a browser or office application was opened at the same time. Mirial Softphone can be quite CPU intensive and appears to adapt to a lack of CPU resource by downgrading the video format at one end of the call (which causes less work for both ends of the call).
A number of experienced videoconferencing equipment users commented on the exceptional quality of conferences involving Mirial Softphone and more then once the comment was made that the quality of image and audio being received at the far end was as good as – and even better than – some dedicated hardware CODECs. This is high praise indeed coming from videoconferencing experts commenting on a desktop application. When using a USB microphone and speaker set, the Logitech® Webcam Pro 9000 and the Windows Vista PC in full screen mode, results were generally excellent, and it felt as though you were sitting in front of a dedicated unit rather than a PC running a software videoconferencing application.
During early use of the software, some video pulsing was experienced, where the picture juddered or appeared to refocus intermittently. However, following software upgrades this problem went away.
When reviewing the results of the tests, it should be borne in mind, that the Windows XP PC also uses a low cost (entry-level) camera, whereas the Windows Vista PC has a good quality, top-of-the-range camera. This allowed the testing of the differences between a cash-strapped environment and one where a little more budget is available.
Windows Vista PC Tests:
Medium close-up, still (audio muted) – the picture quality was very good, although the picture was a bit blurred, especially at 256kbps.
Close up, head shaking (audio muted) – the picture quality was still good at higher speeds, although the picture was blurred and blocky at 256kbps.
Long view, arms waving (audio muted) –at full speed the quality was very good, but there was noticeable blockiness and blurriness at 512kbps and especially at 256kbps.
Medium close-up, still, with audio – even at full speed, lip synchronisation was observed occasionally to be noticeably out of sync, but other than that the quality was rated good, although still with noticeable blurriness at ADSL speeds.
Close up, head shaking, with audio –very good at unlimited bandwidth and 512kbps, but not so good at 256kbps, with blockiness, blur and poor lip-sync experienced.
Long view, arms waving, with audio –overall good results, especially at high speed. At lower speeds a blurred image is the main complaint, with disruptive levels of poor lip-sync, blur and blocks on screen experienced at 256kbps. This test is pushing the limits and the results reflect that.
Windows XP PC Tests:
Medium close-up, still (audio muted) – although good quality the colour accuracy was out and there was some blurring.
Close up, head shaking (audio muted) – blockiness, blurriness, poor colour; the system did not cope with movement very well.
Long view, arms waving (audio muted) – blocky, blurry and jerky at every speed; CPU and network overload warnings were seen at one end during this test. This test at 256kbps produced the lowest scores of all.
Medium close-up, still, with audio – not too bad at unlimited bandwidth, but poorer at 256kbps, and, at that speed, some audio clipping and drop-outs were experienced.
Close up, head shaking, with audio –not too bad overall, but with scores slipping for all factors at 256kbps.
Long view, arms waving, with audio –this didn't rate as poorly as the same test without audio, but blur and poor colour were apparent in every test, and all indicators were consistently poor at 256kbps. Again, the audio quality was affected at 256kbps during this test.
During tests made at 256kbps, the software seemed to cope by sending ever less information in the video format. The format could fluctuate during the same call to less than QCIF (QCIF is resolution at 176x144 pixels, but at one point 128x96 pixels was being sent from one end). Opening a browser or other software while connected at 256kbps sometimes affected the audio quality of the call (as well forcing lower image resolution). The browser or other software also went very slowly. Attempts to share the browser or software using H.239 resulted in virtually unusable audio quality. In this scenario, it would probably be advisable to stop sending video.
Additional tests
The above tests were made using the default video format settings (i.e. 4CIF, 720p and 1080p not enabled). Some additional, ad hoc testing was conducted to test the 4CIF and 720p video formats, and calls were made using these settings quite successfully using the Windows Vista PC, between Mirial Softphone PCs and between a Mirial Softphone PC and various other endpoints. When the Windows XP PC was used, it was far more difficult to get a consistent call, with the format changing on the fly in the light of other user activity on the PC or when faced with a low bandwidth connection (or both). On occasions it seemed that the video format was dropped to a lower resolution for no apparent reason, almost randomly. See appendix 1 for the precise specifications of each PC.
A test was done using speakers and the microphone on the web cam. Acoustic echo cancellation was enabled for this test and, while the audio both ways was not as good as it was when using headphones, it was still perfectly usable. It proved possible to change audio configuration (audio inputs and outputs) during a call without ending the call.
When judging the test scores in Appendix 2, the reader should bear in mind that it is fairly unusual to wave arms and move around a great deal while in a personal videoconference call, so perhaps the most important tests are the medium close-up, still, with audio tests, which most closely resemble the 'talking heads' scenario of personal videoconferences.
Conclusions
Mirial Softphone is a very good piece of videoconferencing software. As mentioned above, many experienced users thought the video and audio quality from Mirial Softphone, using a good PC and camera, were favourably comparable to CODEC equipment costing thousands of pounds. It is true that you are pushing things if you want to do much more with that PC while in a conference call, and doing so will usually cause the quality of the video format to drop, but when being used as a dedicated videoconferencing unit, with the remote image on full screen, it does feel like a dedicated hardware CODEC.
This fully standards compliant software has a high standard of interoperability and it worked with every other H.323 compliant endpoint that we used in the testing process. It also interoperated with the JVCS and would have no problem passing a JVCS QA test. It also offers SIP and so should inter-work happily in a Voice over IP (VoIP) environment.
It would be a very good piece of videoconferencing software even without the additional extras that come with it – the recording facility, desktop sharing and the ability to host three-way meetings. These are the icing on the cake.
However, the software did struggle when installed on the Windows XP PC. In order to get the best results it is necessary to install the software on as powerful a PC as possible, with a good quality camera. In testing, we deliberately kept the host PC 'clean' of extra programmes. Even the amount of software installed seemed to make a difference to test results. If the user tries to do anything else with the PC while in a call, it will affect the call quality, particularly the video format.
So, if Mirial Softphone is to be purchased and used with best results, it should be on a modern PC, with as little else as possible installed or running on it. Use with the defaults – and not taking advantage of the HD capabilities – allows the user to do more on the PC at the same time, without affecting call quality and still gives a very good videoconferencing experience.
Pros:
- turns a PC into a videoconferencing unit effectively and affordably
- good quality SD
- HD quality possible, given a modern fast PC and enough bandwidth
- good standards support, wide interoperability
- compatible with JVCS
- recording and export facility
- hosts a three-way call
Cons:
- user interface could be improved
- needs a fast modern PC to use all features
- no H.460 for firewall traversal servers
Appendix 1 – Description of the PCs and cameras used for the testing
During all the tests, either the two Windows Vista PCs or the two Windows XP PCs were used together, to ensure that the specifications of the PCs and monitors used at each end of each test were exactly the same. Plantronics DSP 400 foldable USB headphones with attached microphones attached were used at both ends, with both machines.
The PC specifications are as follows:
Windows Vista PC
The newer of the two test PCs used for evaluating desktop videoconferencing software for VTAS was purchased in the spring of 2009, and is a Dell™ OptiPlex™ 360 series PC. It has an Intel® Core™2 Duo processor E7400 running at 2.80GHz. It has 4GB of Random Access Memory (RAM) and a 150GB New Technology File System (NTFS) hard drive. It is running Windows Vista Business Edition and has a Windows Experience Rating of 3.5.
The attached USB camera is a Logitech® Webcam Pro 9000 – a near top-of-the-range camera.
This system is intended to replicate an average priced typical PC purchased in 2009 to be used for videoconferencing.
Windows XP PC
The older PC used for VTAS evaluations was purchased in 2005 and is intended to replicate what might typically be found in a secondary or primary school, or the home environment. It is a Dell™ Optiplex™ 170L series PC, with an Intel® Pentium™4 processor running at 2.80GHz. It has 512MB of RAM and is running Windows XP professional, with service packs 2 and 3 installed.
The attached USB camera is a Logitech® QuickCam® Connect™ E2500 – an entry level USB camera.
Appendix 2 – Mirial Softphone H.323 interoperability tests
Point-to-point dialling
The test calls listed below were made without the use of a Multipoint Control Unit (MCU) (i.e. they were point-to-point calls). However, they were made using a TANDBERG Video Communication Server (VCS) Control and TANDBERG Expressway™ acting as gatekeeper and firewall traversal respectively. All possible video and audio encodings and formats were enabled on the Mirial Softphone used for the tests. Bandwidth was set to 1920kbps to try to encourage the two ends of the call to negotiate the best possible encoding and formats. Table 1 shows the results experienced. These are shown as an illustrative guide and only relate to the set of tests made during this evaluation. Other testers using alternative hardware and different networks might experience different results.
Table 1 – Point-to-point interoperability testing
Remote Equipment |
Video encoding |
Video format transmitted |
Video format received |
Audio encoding |
Quality notes |
TANDBERG MXP 6000 |
H.264 |
4CIF |
720p |
G.711 |
Audio is not so good with this protocol |
Polycom® VSX® 8000 |
H.263 |
4CIF |
CIF |
G.722.1c.48 |
Better audio |
Mirial Softphone |
H.264 |
4CIF |
4CIF |
G.722 |
Able to connect at 720p but not always able to keep that quality throughout the call |
Polycom® PVX® |
H.264 |
CIF |
CIF |
G.711 |
Call made at 384kbps |
LifeSize ® Room™ |
H.264 |
720p |
720p |
Siren 14 (48kbps) |
This was the best sustained quality, both video and audio, of all of the tests made |
Possible video formats available in Mirial Softphone (depending on hardware and network). 720p and 1020p are HD and will produce a widescreen (16:9) aspect ratio |
|
Format |
Size (in pixels) |
SQCIF |
128x96 |
QCIF |
176x144 |
CIF |
352x288 |
4CIF |
704x576 |
720p |
1280x720 |
1080p |
1920x1080 |
When testing through an MCU, the quality was generally higher in both directions. Even a call between two Mirial Softphones seems to behave better and give a more constant higher quality when there was an MCU between the two participants. During a call involving a Codian MCU, both ends transmitted and received 720p and G.711 throughout the call.
With this MCU brokering the call, better results were obtained than indicated in the table above.
Table 2 – Interoperability testing with a Codian HD MCU between the endpoints
Remote equipment |
Video encoding |
Video format transmitted |
Video format received |
Audio encoding |
Quality notes |
TANDBERG MXP 6000 |
H.264 |
720p/15fps[1] |
720p/25fps |
G.722.1c |
Sharp picture, slightly jerky movement |
Polycom® VSX® 7000 |
H.264 |
4CIF/10fps |
640x448 (ED[2]) |
G.722.1c |
As above |
Mirial Softphone |
H.264 |
720p |
720p |
G.722 |
Video encoding stable throughout the call |
Polycom® HDX® 9000 |
H.264 |
720p |
720p |
G.722 |
The remote picture froze. It was necessary to bring the call down to 4CIF |
Polycom |
H.261 |
CIF |
CIF |
G.711 |
Worked, but poor quality |
One test was made using a Polycom® MGC™ (Standard definition) MCU. This was between the Mirial Softphone and a Polycom HDX 9000. The video encoding negotiated was H.263, G.711 audio, and a CIF image in both directions.
An ad hoc test was also made to a Mac running XMeeting. The call did complete, but only after reducing the optional video CODECs to the oldest and poorest quality video encoding, H.261.
Appendix 3 – Full results of the evaluation testing of Mirial Softphone
Tests were conducted at the different bandwidths, from both ends of the call, and the call quality was rated by the receiving end in each case. The following potential impairments were considered and rated by the evaluators, and marked on a scale of one to five, where:
1 = imperceptible;
2 = slight or occasional perceptible impairment;
3 = perceptible impairment;
4 = constant (significant) impairment;
5 = impairment to a disruptive degree.
The actual tests carried out were:
- V11 Medium close-up, still (audio muted)
- V12 Close up, head shaking (audio muted)
- V13 Long view, arms waving (audio muted)
- V14 Medium close-up, still, with audio
- V15 Close up, head shaking, with audio
- V16 Long view, arms waving, with audio
The quality issues considered (subjective video impairments tested) during these test calls were as follows:
- LS Lip synchronisation
- BLK Block distortion (tiling)
- BLR Blurring (reduced edge sharpness and spatial detail)
- CLR Colour error
- JRK Jerkiness (distortion of smooth motion)
- OP Object persistence (lagging images from previous frames as faded or outline images)
The following pages of test results should be read in conjunction with the above information.
Individual test scores – Windows Vista PC, full screen, unlimited bandwidth (1920kbps), H.264, CIF
Test reference |
LS |
BLK |
BLR |
CLR |
JRK |
OP |
V11 - GC |
---------- |
1 |
2 |
1 |
N/A |
N/A |
V11 - RG |
---------- |
2 |
2 |
1 |
N/A |
N/A |
V11 - Average |
---------- |
1.5 |
2 |
1 |
N/A |
N/A |
V12 - GC |
---------- |
1 |
1 |
1 |
1 |
1 |
V12 - RG |
---------- |
1 |
2 |
1 |
1 |
1 |
V12 - Average |
--------- |
1 |
1.5 |
1 |
1 |
1 |
V13 - GC |
---------- |
1 |
1 |
1 |
1 |
1 |
V13 - RG |
---------- |
1 |
3 |
1 |
1 |
1 |
V13 - Average |
--------- |
1 |
2 |
1 |
1 |
1 |
V14 - GC |
2 |
1 |
2 |
1 |
1 |
1 |
V14 - RG |
3 |
1 |
2 |
1 |
1 |
1 |
V14 - Average |
2.5 |
1 |
2 |
1 |
1 |
1 |
V15 - GC |
2 |
1 |
1 |
1 |
1 |
1 |
V15 - RG |
2 |
1 |
2 |
1 |
1 |
1 |
V15 - Average |
2 |
1 |
1.5 |
1 |
1 |
1 |
V16 - GC |
1 |
1 |
1 |
1 |
2 |
1 |
V16 - RG |
3 |
1 |
3 |
1 |
1 |
1 |
V16 - Average |
2 |
1 |
2 |
1 |
1.5 |
1 |
Individual test scores – Windows Vista PC, full screen, bandwidth limited to
512kbps, H.264, CIF
Test reference |
LS |
BLK |
BLR |
CLR |
JRK |
OP |
V11 - GC |
---------- |
1 |
2 |
1 |
N/A |
N/A |
V11 - RG |
---------- |
1 |
3 |
1 |
N/A |
N/A |
V11 - Average |
---------- |
1 |
2.5 |
1 |
N/A |
N/A |
V12 - GC |
---------- |
1 |
2 |
1 |
2 |
1 |
V12 - RG |
---------- |
1 |
3 |
1 |
1 |
1 |
V12 - Average |
--------- |
1 |
2.5 |
1 |
1.5 |
1 |
V13 - GC |
---------- |
3 |
2 |
2 |
1 |
1 |
V13 - RG |
---------- |
1 |
4 |
1 |
1 |
1 |
V13 - Average |
--------- |
2 |
3 |
1.5 |
1 |
1 |
V14 - GC |
1 |
1 |
1 |
1 |
1 |
1 |
V14 - RG |
2 |
1 |
3 |
1 |
1 |
1 |
V14 - Average |
1.5 |
1 |
2 |
1 |
1 |
1 |
V15 - GC |
1 |
3 |
2 |
1 |
1 |
1 |
V15 - RG |
2 |
1 |
3 |
1 |
1 |
1 |
V15 - Average |
1.5 |
2 |
2.5 |
1 |
1 |
1 |
V16 - GC |
1 |
3 |
2 |
2 |
1 |
1 |
V16 - RG |
3 |
2 |
4 |
1 |
1 |
1 |
V16 - Average |
2 |
2.5 |
3 |
1.5 |
1 |
1 |
Individual test scores – Windows Vista PC, full screen, bandwidth limited to
256kbps, H.264, CIF
Test reference |
LS |
BLK |
BLR |
CLR |
JRK |
OP |
V11 - GC |
---------- |
1 |
2 |
1 |
N/A |
N/A |
V11 - RG |
---------- |
1 |
4 |
1 |
N/A |
N/A |
V11 - Average |
---------- |
1 |
3 |
1 |
N/A |
N/A |
V12 - GC |
---------- |
4 |
3 |
2 |
1 |
1 |
V12 - RG |
---------- |
3 |
5 |
2 |
1 |
1 |
V12 - Average |
--------- |
3.5 |
4 |
2 |
1 |
1 |
V13 - GC |
---------- |
4 |
3 |
2 |
1 |
1 |
V13 - RG |
---------- |
4 |
5 |
2 |
1 |
1 |
V13 - Average |
--------- |
4 |
4 |
2 |
1 |
1 |
V14 - GC |
2 |
1 |
2 |
1 |
1 |
1 |
V14 - RG |
2 |
2 |
4 |
2 |
1 |
1 |
V14 - Average |
2 |
1.5 |
3 |
1.5 |
1 |
1 |
V15 - GC |
3 |
3 |
2 |
2 |
1 |
1 |
V15 - RG |
3 |
3 |
4 |
2 |
1 |
1 |
V15 - Average |
3 |
3 |
3 |
2 |
1 |
1 |
V16 - GC |
3 |
5 |
4 |
2 |
2 |
1 |
V16 - RG |
5 |
5 |
5 |
2 |
2 |
1 |
V16 - Average |
4 |
5 |
4.5 |
2 |
2 |
1 |
Individual test scores – Windows XP PC, uncapped bandwidth (1920kbps), H.264, CIF
Test reference |
LS |
BLK |
BLR |
CLR |
JRK |
OP |
V11 - GC |
---------- |
1 |
2 |
2 |
N/A |
N/A |
V11 - RG |
---------- |
2 |
3 |
3 |
N/A |
N/A |
V11 - Average |
---------- |
1.5 |
2.5 |
2.5 |
N/A |
N/A |
V12 - GC |
---------- |
4 |
3 |
3 |
2 |
1 |
V12 - RG |
---------- |
2 |
4 |
3 |
1 |
1 |
V12 - Average |
--------- |
3 |
3.5 |
3 |
1.5 |
1 |
V13 - GC |
---------- |
3 |
4 |
3 |
2 |
1 |
V13 - RG |
---------- |
2 |
4 |
3 |
3 |
2 |
V13 - Average |
--------- |
2.5 |
4 |
3 |
2.5 |
1.5 |
V14 - GC |
1 |
1 |
2 |
2 |
1 |
1 |
V14 - RG |
1 |
2 |
3 |
3 |
2 |
1 |
V14 - Average |
1 |
1.5 |
2.5 |
2.5 |
1.5 |
1 |
V15 - GC |
1 |
4 |
4 |
2 |
1 |
1 |
V15 - RG |
2 |
2 |
3 |
3 |
2 |
1 |
V15 - Average |
1.5 |
3 |
3.5 |
2.5 |
1.5 |
1 |
V16 - GC |
2 |
4 |
3 |
3 |
2 |
1 |
V16 - RG |
3 |
3 |
4 |
3 |
2 |
2 |
V16 - Average |
2.5 |
3.5 |
3.5 |
3 |
2 |
1.5 |
Individual test scores – Windows XP PC, 512kbps, H.264, varied video formats
Test reference |
LS |
BLK |
BLR |
CLR |
JRK |
OP |
V11 - GC |
---------- |
1 |
3 |
2 |
N/A |
N/A |
V11 - RG |
---------- |
1 |
3 |
3 |
N/A |
N/A |
V11 - Average |
---------- |
1 |
3 |
2.5 |
N/A |
N/A |
V12 - GC |
---------- |
4 |
3 |
2 |
2 |
1 |
V12 - RG |
---------- |
2 |
3 |
3 |
2 |
1 |
V12 - Average |
--------- |
3 |
3 |
2.5 |
2 |
1 |
V13 - GC |
---------- |
3 |
4 |
3 |
2 |
2 |
V13 - RG |
---------- |
2 |
4 |
3 |
2 |
1 |
V13 - Average |
--------- |
2.5 |
4 |
3 |
1 |
1.5 |
V14 - GC |
2 |
2 |
3 |
2 |
1 |
1 |
V14 - RG |
2 |
2 |
3 |
3 |
2 |
1 |
V14 - Average |
2 |
2 |
3 |
2.5 |
1.5 |
1 |
V15 - GC |
4 |
3 |
4 |
2 |
1 |
2 |
V15 - RG |
2 |
2 |
3 |
3 |
2 |
1 |
V15 - Average |
3 |
2.5 |
3.5 |
2.5 |
1.5 |
1.5 |
V16 - GC |
2 |
2 |
4 |
3 |
1 |
1 |
V16 - RG |
3 |
2 |
3 |
3 |
2 |
1 |
V16 - Average |
2.5 |
2 |
3.5 |
3 |
1.5 |
1 |
Individual test scores – Windows XP PC, 256kbps, H.264, varied video formats
Test reference |
LS |
BLK |
BLR |
CLR |
JRK |
OP |
V11 - GC |
---------- |
1 |
2 |
2 |
N/A |
N/A |
V11 - RG |
---------- |
2 |
4 |
3 |
N/A |
N/A |
V11 - Average |
---------- |
1.5 |
3 |
2.5 |
N/A |
N/A |
V12 - GC |
---------- |
2 |
4 |
2 |
2 |
1 |
V12 - RG |
---------- |
3 |
4 |
3 |
2 |
1 |
V12 - Average |
--------- |
2.5 |
4 |
2.5 |
2 |
1 |
V13 - GC |
---------- |
4 |
4 |
2 |
2 |
2 |
V13 - RG |
---------- |
3 |
4 |
3 |
5 |
3 |
V13 - Average |
--------- |
3.5 |
4 |
2.5 |
3.5 |
2.5 |
V14 - GC |
3 |
1 |
2 |
2 |
2 |
1 |
V14 - RG |
3 |
2 |
4 |
3 |
2 |
1 |
V14 - Average |
3 |
1.5 |
3 |
2.5 |
4 |
1 |
V15 - GC |
3 |
4 |
4 |
3 |
3 |
2 |
V15 - RG |
2 |
4 |
4 |
3 |
2 |
1 |
V15 - Average |
2.5 |
4 |
4 |
3 |
2.5 |
1.5 |
V16 - GC |
4 |
4 |
4 |
3 |
2 |
2 |
V16 - RG |
4 |
4 |
4 |
3 |
3 |
1 |
V16 - Average |
4 |
4 |
4 |
3 |
2.5 |
1.5 |
Audio drop out was experienced during tests V15 and V16.