Evaluation of Skype 4.0

Desktop Videoconferencing Software Evaluations

Skype™ 4.0

Introduction

This report is one of an ongoing series of evaluations of desktop videoconferencing software that is being produced by the Video Technology Advisory Service (VTAS). The tests and evaluation have been conducted in accordance with the principles and procedures set out in the Desktop Evaluations Overview.

Skype is a mature product that has been available as a free download since 2004. It should be noted that the application uses completely proprietary (i.e. not standards based) methods to make calls. Originally the product was audio-only, 'Internet Phone Call' software. Version 4.0 also includes the ability to make video calls. It offers a simple to use means of live audio (or audio and video) conversation with friends and family, as well as the potential for educational and business use. Its most attractive features are that it is free and easily available.

The features of Skype that raised concerns in the past – use of bandwidth and encrypted tunnels – have been addressed by Skype in recent releases.

Basic Details

Application: Skype

Software Version Tested: 4.0.0.226

Test Dates: June/July 2009

Testers: Geoff Constable and Roger Gachago, with help from Matt Collins.

Skype allows Personal Computer (PC) and Mac® users to make free video and audio calls between devices on the Internet. It also offers the ability to make charged voice calls to non-Internet telephones. The client also offers file transfer and text chatting. Third party add-ons extend the basic functionality by adding games and other applications (including desktop sharing). New software releases are fairly frequent, and there has been at least one major software release since this evaluation.

Costs

Skype is completely free to download and use. The downloaded version is the full version – there is no “Pro” version with additional functions available to purchase. The company makes money by offering enhanced, additional functionality for extended use beyond Internet-only calls. There are additional charges for dialling through to a non-Internet telephone (i.e. a land-line or mobile). These additional services are paid for by a pay-as-you-go (credit) agreement or by a flat-rate subscription.

Standards and Security

Skype does not reveal what standards it uses for call setup, or for audio and/or video encoding. Skype for SIP Beta software is in preparation. The application uses port 80 for communication, the same as the HyperText Transfer Protocol (HTTP World Wide Web). This means that it should work 'out of the box' – without the need for any firewall changes. If the network manager has restricted use of port 80 to HTTP then it will be necessary to relax this before Skype will work through the firewall.

Skype uses standards-based, Advanced Encryption Standard (AES) encryption algorithms to encrypt all calls (including file transfers and chat sessions), although calls to the Public Switched Telephone Network (PSTN) are not encrypted once the data leaves the Internet. It also uses digital certificates to authenticate the integrity of users at each end.

Installation

To install Skype, visit www.skype.com and select the Download tab. Once the installation file has been downloaded, run it to install Skype. The download file at version 4.0 was 20 MB. The process is very simple and requires little or no configuration. Skype automatically detects the audio and camera equipment installed on the PC. If you have various options for inputting or outputting audio or video, Skype will select one of these as the default equipment. If the default selection is not the equipment that you wish to use, there should be a drop down list containing all other equipment options, and it is possible to select the item you wish to have selected by default from these options. The entire process takes less than five minutes.

Once you have downloaded and started the installation, you will see a welcome screen:

Figure 1: Skype Install screen

Before you can start to use Skype you have to create a Skype account. This will enable you to find other Skype users and make calls.

Figure 2: Create a new Skype account

When you have identified a unique name for your account, and registered successfully, you will see the welcome screen confirming installation. You will also be offered the opportunity to make a test call to Skype. This is useful for confirming you have selected the correct peripherals and that you will be able to communicate with people when you actually come to use Skype for real.

Figure 3: Welcome screen

Documentation

No additional documentation is supplied with the software (no readme file or release notes, for example). However, there is extensive help and support available, both at Skype's main site (www.skype.com) and also the support site (https://support.skype.com). As well as a frequently asked questions section in a searchable format, there are also links to online discussion forums, which offer more detailed support.

Online guides start up after you have installed Skype that take you through basic usage. All in all, although there is no written material provided with the application, the online help and support are very good and are easy to use.

One problem in initial usage is encountered if you do not know what your friends' Skype accounts are called. You can search all registered users by their real names, but, in the author's experience, that doesn't necessarily mean the search will be successful. When you want to Skype with someone, it would probably be a good idea to exchange Skype account details first.

User Interface

The undoubted success of Skype is probably down to three important factors: it is free, of good quality and easy to use. The interface is intuitive and easy to configure. For example, it is possible to adjust the size of the picture-in-picture as you would any other window. It is also possible to have the self-view window as a separate window.

Buttons and menu items are generally intuitive. The actions available from the various buttons on the interface have tooltips (small pop-up windows with descriptive text, which appear when the mouse is hovered over an icon or button), which also help the novice user. The interface is clear and uncomplicated and can be configured to look as you wish. For instance, under the View; Compact View menu it is possible to separate the two different functional elements of the interface: the Contact, Directory and Calling interface and the text, audio or video call that is in progress. This allows more of the PC screen to be devoted to the call than if the default view is retained (when both these elements of the interface appear as a single window).

Figure 4: Default view with call control element and video window unified

Figure 5: Compact view (with video window separated and call control window minimised)

Connectivity

It was not necessary to make any changes to the firewall on the local PC, or the organisational firewall, in order to use Skype. It was possible to both initiate and receive calls with no need to disturb the firewall manager.

It was not necessary to know any technical information or undertake any configuration in order to start using Skype. There was no need to supply details of the network, like IP addresses of contacts or servers, or any other technical or network information.

At the time of writing, Skype cannot be used with the JANET Videoconferencing Service (JVCS) or any other standards-based, IP videoconferencing equipment and there are no gateways available between Skype and standards-based (Session Initiation Protocol (SIP), H.323) equipment. However, there is a Skype for SIP Beta program, which indicates that Skype may interoperate with SIP at some point in the future.

Multi-way video calls are not possible: if a point-to-point video call is extended in an ad hoc manner to include a third (or more) participants, the call returns to audio-only for all participants. The multi-way capacity was tested at uncapped bandwidth with three and four participants, with no detrimental effect on the audio quality. The Skype documentation states that multi-way capacity can extend to up to 24 participants in the same call; however, this feature was not tested beyond four participants. The terminology used by Skype changes from a 'call' to a 'conference' when there are more than two participants.

Call Procedures

In order to make a call you need to have someone on your contact list. You can add people manually, if they have told you their Skype account name, or you can search for them. This is only necessary the first time that you call them; thereafter their Skype ID will appear in your contacts list. Once you have your contact’s details you can add them to your contacts list, although they will be asked to allow this before you can call them. Similarly, you cannot be added to another user's contacts list without your approval.

The main Skype interface displays the number of people currently online and this varied during the testing period between 14 million and 16 million (to the nearest million). The Find your friends function still manages to be very fast and will find people with the personal details you have cited very quickly. There will usually be a short list of people that have the same name as the person you are looking for, but it should be possible to identify your contact from their location.

To call someone you select them in your contacts list and click Call or Video call. It is possible to start the call in audio only and then add video. In fact video can be selected to be sent to every call by default, or can be switched on or off during a call at will.

Data Sharing

Skype does not include any data sharing functionality. It does include a number of third-party add-ons, including games and desktop sharing applications, but as these are third-party add-ons, they were not included in this evaluation.

Skype does include an instant messaging function, which sends text messages that are logged in an activity history. During a multi-way call it is possible to have a separate text conversation with an individual by selecting that individual in the Contacts tab.

Quality Testing

These tests were performed on the same equipment used for all VTAS Desktop testing in 2009. A 2005 Windows XP PC and a 2009 Windows Vista PC were chosen to allow testing to be carried out on both the type of equipment commonly found in use in homes and schools and up-to-date equipment. The specification of the equipment is described in detail in Appendix 1.

Audio Tests

All testing was performed in accordance with the outlines described in the Desktop Evaluations Overview document (REF). The standard equipment (i.e. headsets with microphones) was used for all the audio and video tests. For some other calls, a set of PC speakers was used at one end of the call. During these calls the audio play out was fine at the end with the speakers, but audio clipping and drop-outs could be heard at the end that was using a headset, suggesting that the echo cancellation was not coping with the speakers. The microphone used was the camera's built-in microphone (and the speakers were positioned quite close to it on a desk), so this may be a hardware issue rather than anything related to Skype.

Tests were conducted at 256kbps, 512kbps, and uncapped bandwidth. These are roughly comparable to a poor ADSL connection, a good ADSL connection and an office Local Area Network (LAN) respectively. Tests were repeated with video muted and video enabled.

Audio levels were found to be adequate in all tests, audio quality was acceptable in all tests, and echo cancellation was acceptable in all cases. With video enabled at the lower bandwidths, the audio quality (while still acceptable and usable) did become more tinny, and double talk (both participants speaking at the same time) was noticeably poorer at the ADSL speeds. This was not apparent at uncapped bandwidth. It was also apparent that audio quality was better when there was no accompanying video at ADSL speeds. Audio quality was noticeably poorer on the Windows XP PC, but was still very good; this was particularly noticeable during the double-talk test.

Video Tests

As might be expected, results varied with between the Windows Vista PC and the Windows XP PC. The differences were exacerbated by the use of the cheaper web cam with the Windows XP PC.

Windows Vista PC tests:

Medium close-up, still (audio muted) - as might be expected with little movement and no audio, the scores were consistently good on all criteria for these tests, across all bandwidths.

Close up, head shaking (audio muted) - only occasional and slight imperceptions were noticed at all test bandwidths.

Long view, arms waving (audio muted) - the waving arms caused some consistently perceptible jerkiness and blur at all bandwidths, particularly the lower bandwidths.

Medium close-up, still, with audio - very little perceptible fault with this test at all bandwidths.

Close up, head shaking, with audio - the results with uncapped bandwidth were still quite good, although the movement did mean that some jerkiness was consistently perceptible, and lip-synchronisation was noticeably poorer. At the lower bandwidths, lip-synchronisation and jerkiness became consistently and constantly, significantly impaired.

Long view, arms waving, with audio - results showed constant perceptible impairments with this usage, even at unlimited bandwidth. Lip-synchronisation was particularly poor at the lower bandwidths.

Windows XP PC tests:

Those tests that involved movement, such as head shaking and arm waving, showed perceptible jerkiness and blurring at every bandwidth. Lip-synchronisation was poor in all the tests with audio. As might be expected, results were worse at lower bandwidths. During these tests a green hue on the image was apparent occasionally (this occurred at both ends). This was resolved by stopping and re-starting the video being sent.

Medium close-up, still (audio muted) – probably due to the cheaper camera in use with this system, the picture was noticeably blurry, and the colour not as good as with the Windows Vista PC.

Close up, head shaking (audio muted) – jerkiness was a problem with movement, particularly at 256kbps.

Long view, arms waving (audio muted) – movement caused the image to be constantly blurred and jerky, even at uncapped bandwidth.

Medium close-up, still, with audio – despite the still image, blur, jerkiness and poor lip-synchronisation were all perceptible in the facial movement of speech.

Close up, head shaking, with audio – lip synchronisation was very poor and this was noticeable throughout these tests at every bandwidth.

Long view, arms waving, with audio – again, blur, jerkiness and poor lip-synchronisation were noticeable and constant.

Conclusions

It should be remembered that the prime use of Skype is as a personal communication tool. Some organisations have tried to use it as a room application, by pointing the camera at a group of people and projecting the desktop, but this is misguided, as the application cannot cope with this level of movement. However, for the purpose of desktop communication, it is an excellent piece of software, especially considering that it comes for free. These tests show that it performs best with a more up to date PC and better quality camera, and its faults are more noticeable with cheaper and older equipment.

The fact that it is used by so many people throughout the world demonstrates that it can perform acceptably when on limited bandwidth, similar to that experienced over home ADSL broadband. However, attempting to use Skype when the image is full of movement stretches the software; this is even more apparent on an older PC. Having said that, it is worth pointing out that Skype is primarily an audio tool, and throughout all testing the audio quality was very good – and usually excellent. It was slightly more tinny when using the Windows XP PC at lower bandwidths, but was still perfectly usable, and comfortable, for long periods.

For those who want a free to use, acceptable quality solution that can be installed quickly and requires no network or firewall changes, Skype is ideal. The only significant drawbacks are that it does not cope well with a lot of movement and, more importantly, it is a proprietary solution that does not interoperate with the JVCS or any standards based videoconferencing equipment such as those supporting H.323 or SIP.

Appendix 1 – Description of the PCs and cameras used for the testing

During all the tests, either the two Windows Vista PCs or the two Windows XP PCs were used together, to ensure that the specifications of the PCs and monitors used at each end of each test were exactly the same. Plantronics DSP 400 foldable USB headphones with attached microphones attached were used at both ends, with both machines.

The PC specifications are as follows:

Windows Vista PC

The newer of the two test PCs used for evaluating desktop videoconferencing software for VTAS was purchased in the spring of 2009, and is a Dell™ OptiPlex™ 360 series PC. It has an Intel® Core™2 Duo processor E7400 running at 2.80GHz. It has 4GB of RAM and a 150GB NTFS hard drive. It is running Windows Vista Business Edition and has a Windows Experience Rating of 3.5.

The attached USB camera is a Logitech® Webcam Pro 9000 – a near top-of-the-range camera.

This system is intended to replicate an average priced typical PC purchased in 2009 to be used for videoconferencing.

Windows XP PC

The older PC used for VTAS evaluations was purchased in 2005 and is intended to replicate what might typically be found in a secondary or primary school, or the home environment. It is a Dell™ Optiplex™ 170L series PC, with an Intel® Pentium™4 processor running at 2.80GHz. It has 512MB of RAM and is running Windows XP professional, with service packs 2 and 3 installed.

The attached USB camera is a Logitech® QuickCam® Connect™ E2500 – an entry level USB camera.

Appendix 2 – Full results of the evaluation testing on Skype

Tests were conducted at the different bandwidths, from both ends of the call, and the call quality was rated by the receiving end in each case. The following potential impairments were considered and rated by the evaluators, and marked on a scale of one to five, where:

1 = imperceptible

2 = slight or occasional perceptible impairment

3 = perceptible impairment

4 = constant (significant) impairment

5 = impairment to a disruptive degree

The actual tests carried out were:

V11 Medium close-up, still (audio muted)
V12 Close up, head shaking (audio muted)
V13 Long view, arms waving (audio muted)
V14 Medium close-up, still, with audio
V15 Close up, head shaking, with audio
V16 Long view, arms waving, with audio

The quality issues considered (subjective video impairments tested) during these test calls were as follows:

LS Lip synchronisation
BLK Block distortion (tiling)
BLR Blurring (reduced edge sharpness and spatial detail)
CLR Colour error
JRK Jerkiness (distortion of smooth motion)
OP Object persistence (lagging images from previous frames as faded or outline images)
SCR Scene cut response (i.e. time to build up the new image)

The following pages of test results should be read in conjunction with the above information.

Individual test scores – Windows Vista PC, unlimited bandwidth

Test reference	LS	BLK	BLR	CLR	JRK	OP
V11 - GC	----------	2	1	2	N/A	N/A
V11 - RG	----------	2	2	2	N/A	N/A
V11 – Average	----------	2	1.5	2	N/A	N/A
V12 - GC	----------	3	3	1	3	1
V12 - RG	----------	2	2	2	3	1
V12 - Average	---------	2.5	2.5	1.5	3	1
V13 - GC	----------	2	4	1	3	2
V13 - RG	----------	2	3	2	3	1
V13 - Average	---------	2	3.5	1.5	3	1.5
V14 - GC	2	2	1	1	2	-
V14 - RG	1	2	2	2	2	-
V14 - Average	1.5	2	1.5	1.5	2	-
V15 - GC	3	2	2	1	2	1
V15 - RG	2	2	2	2	3	1
V15 - Average	2.5	2	2	1.5	2.5	1
V16 - GC	2	2	3	1	3	1
V16 - RG	3	2	3	2	4	1
V16- Average	2.5	2	3	1.5	3.5	1

Individual test scores – Windows Vista PC, bandwidth limited to 512 kbps

Test reference	LS	BLK	BLR	CLR	JRK	OP
V11 - GC	----------	1	1	1	N/A	N/A
V11 - RG	----------	2	2	2	N/A	N/A
V11 – Average	----------	1.5	1.5	1.5	N/A	N/A
V12 - GC	----------	1	3	1	2	1
V12 - RG	----------	2	2	2	3	1
V12 - Average	---------	1.5	2.5	1.5	2.5	1
V13 - GC	----------	2	4	1	3	1
V13 - RG	----------	2	2	2	4	1
V13 - Average	---------	2	3	1.5	3.5	1
V14 - GC	2	1	1	1	2	1
V14 - RG	2	2	2	2	2	1
V14 - Average	2	1.5	1.5	1.5	2	1
V15 - GC	3	2	3	1	3	1
V15 - RG	4	2	2	2	4	1
V15 - Average	3.5	2	2.5	1.5	3.5	1
V16 - GC	3	3	4	1	4	1
V16 - RG	5	2	2	2	4	1
V16- Average	4	2.5	3	1.5	4	1

Individual test scores – Windows Vista PC, bandwidth limited to 256 kbps

Test reference	LS	BLK	BLR	CLR	JRK	OP
V11 - GC	----------	2	1	1	N/A	N/A
V11 - RG	----------	2	2	2	N/A	N/A
V11 – Average	----------	2	1.5	1.5	N/A	N/A
V12 - GC	----------	3	2	1	3	1
V12 - RG	----------	2	2	2	3	1
V12 - Average	---------	2.5	2	1.5	3	1
V13 - GC	----------	3	3	1	2	1
V13 - RG	----------	2	3	2	3	1
V13 - Average	---------	2.5	3	1.5	2.5	1
V14 - GC	2	1	2	1	2	1
V14 - RG	1	2	2	2	1	1
V14 - Average	1.5	1.5	2	1.5	1.5	1
V15 - GC	3	2	3	1	2	1
V15 - RG	4	3	3	2	2	1
V15 - Average	3.5	2.5	3	1.5	2	1
V16 - GC	3	4	3	1	3	1
V16 - RG	4	4	3	3	2	1
V16- Average	3.5	4	3	2	2.5	1

Individual test scores – Windows XP PC, uncapped bandwidth

Test reference	LS	BLK	BLR	CLR	JRK	OP
V11 - GC	----------	2	3	3	N/A	N/A
V11 - RG	----------	2	3	3	N/A	N/A
V11 – Average	----------	2	3	3	N/A	N/A
V12 - GC	----------	2	4	2	4	2
V12 - RG	----------	2	3	3	3	1
V12 - Average	---------	2	3.5	2.5	3.5	1.5
V13 - GC	----------	2	4	3	5	1
V13 - RG	----------	2	3	3	4	1
V13 - Average	---------	2	3.5	3	4.5	1
V14 - GC	3	2	2	2	3	1
V14 - RG	1	2	3	3	2	1
V14 - Average	2	2	2.5	2.5	2.5	1
V15 - GC	5	3	3	2	4	1
V15 - RG	5	3	3	3	4	1
V15 - Average	5	3	3	2.5	4	1
V16 - GC	5	2	4	2	4	1
V16 - RG	5	3	4	3	4	1
V16- Average	5	2.5	4	2.5	4	1

Individual test scores – Windows XP PC, 512 kbps

Test reference	LS	BLK	BLR	CLR	JRK	OP
V11 - GC	----------	1	3	2	N/A	N/A
V11 - RG	----------	2	3	3	N/A	N/A
V11 – Average	----------	1.5	3	2.5	N/A	N/A
V12 - GC	----------	2	4	2	4	1
V12 - RG	----------	2	3	3	3	1
V12 - Average	---------	2	3.5	2.5	3.5	1
V13 - GC	----------	2	5	2	4	1
V13 - RG	----------	2	4	3	3	2
V13 - Average	---------	2	4.5	2.5	3.5	1.5
V14 - GC	4	2	2	2	4	1
V14 - RG	1	2	3	3	2	1
V14 - Average	2.5	2	2.5	2.5	3	1
V15 - GC	5	3	5	3	5	1
V15 - RG	5	3	3	3	4	1
V15 - Average	5	3	4	3	4.5	1
V16 - GC	4	2	4	3	5	2
V16 - RG	2	2	3	3	3	2
V16- Average	3	2	3.5	3	4	2

Individual test scores – Windows XP PC, 256 kbps

Test reference	LS	BLK	BLR	CLR	JRK	OP
V11 - GC	----------	2	2	2	N/A	N/A
V11 - RG	----------	2	3	3	N/A	N/A
V11 – Average	----------	2	2.5	2.5	N/A	N/A
V12 - GC	----------	2	4	3	5	2
V12 - RG	----------	2	3	3	4	1
V12 - Average	---------	2	3.5	3	4.5	1.5
V13 - GC	----------	2	5	3	5	2
V13 - RG	----------	2	4	3	4	1
V13 - Average	---------	2	4.5	3	4.5	1.5
V14 - GC	4	1	2	2	3	1
V14 - RG	3	2	3	3	2	1
V14 - Average	3.5	1.5	2.5	2.5	2.5	1
V15 - GC	5	3	3	2	3	1
V15 - RG	4	2	3	3	3	1
V15 - Average	4.5	2.5	3	2.5	3	1
V16 - GC	4	2	5	3	3	1
V16 - RG	5	3	5	3	3	1
V16- Average	4.5	2.5	5	3	3	1