Device Testing vs. App Testing: Choosing the Right Tool for the Right Job

testingWhen it comes to mobile app testing, it’s critical to use the right tool for the right job. That’s especially true when you’re considering which strategies to use for device testing vs. app testing.

Mobile device testing is important in a limited number of test cases. For example, you will want to test to ensure your app responds properly to a location services prompt of the operating system.

Driving Device Functions

Device control apps enable you to remotely view and interact with the device as though it were in your hands. This includes the ability to wake the device and enter a pass code. You can also access all apps and functions, including messages, calendars, photos and settings.

As a result, you can make sure your mobile app works properly with device functions such as the phone dealer or built-in messaging apps.

Such tests require access to the device home screen and other phone or tablet functions and automation to drive the simple user interface of the device’s built-in apps.

With remote device management solutions, you can remotely access and manipulate a device to perform functions that go beyond the scope of the app, such as pressing the home button or deleting an app.

Taking App Testing Up a Notch

On the other hand, mobile app testing takes testing to another level of seriousness – beyond ringtones, messaging functions and location services. When it comes right down to it, what’s most important is whether customers can use your mobile app for the purpose it was intended – whether that’s using the app to make a hotel reservation or checking to see what items you have in inventory.

Mobile app testing focuses on whether the app actually works. It validates the core business logic of your application, which helps your organization be more competitive and ensures customers are happy.

That’s where object-based instrumentation tools come in. These tools give you a full view of the device and full control of the app.

With an instrumentation approach using an object repository, testers can also manage objects all in one location. So your mobile testing software would automatically recognize the objects, which will make code changes or script updates unnecessary in many cases.

Your Mobile App Testing Arsenal

When it’s time to choose your automated testing tools, it shouldn’t be a matter of choosing between device testing vs. mobile app testing. Both should be part of your mobile application testing arsenal. Make sure your tool set allows your testers to perform both, and you’ll ensure you always have the right tool for the right job.

Mobile Device Testing: Real Devices vs Emulators Pros & Cons

If you’re reading this, you probably already understand the importance of, and why you need mobile device testing to be a part of your mobile testing strategy. If you’d like some tips, here’s a great article on mobile device testing strategy.

So once you have your plan, now what? Well, you’ll need to decide how you’re going to go about testing and with what testing tools.

When it really comes down to it, there’s 2 ways to test. Either on real devices or with an emulator which simulates the software. Each has its pros and cons so let’s explore!

Testing on Real Devices

There’s really no substitute for the real deal. Testing on real android/ios/other devices is the absolute best way to get an idea of how an app works on each type of device. However, there can be some drawbacks.

First, you need to have all those devices to test on. This can be accomplished either by hitting up all of your coworkers and borrowing devices (commonly referred to as BYOD, Bring Your Own Device… Yuck) or by contacting device manufacturers and asking to borrow. If you’re lucky, you’ll be put on a wait list and receive a device to test on… Eventually. Then there’s actual testing part. Manual device testing is crucial, but it is extremely time consuming, especially when testing for the first time to identify and diagnose bugs.

That’s where automated app testing comes in, it’s just very difficult to do on dozens of devices all at once.

Fortunately, there are a handful of mobile device testing solutions out there. For example, device Connect by Mobile Labs allows you to test almost 50 devices at once and automate the testing process on every device simultaneously. You actually receive a physical testing station which can be connected behind your own corporate firewall. Say “bye-bye” to 3rd party security concerns. Mobile Labs also has good reviews too!

This solution really is only meant for serious testers in a larger company. While the equipment may seem expensive, it’s significantly cheaper than purchasing all the devices and paying monthly and it can streamline a process that would take several developers weeks to test, done to just a few developers who can test in a fraction of the time.

Real Devices Pros:

  • Most accurate testing environment
  • Increased security
  • With a tool like device Connect, it can save significant time and money
  • Complete control of the testing process

Real Devices Cons

  • Can be expensive and slow without the right tools
  • Physical devices do take up a little bit of space

Testing on Emulators

For those without the capital resources and companies not large enough to test on mobile devices on a large scale, there’s the emulator solution. Emulators mimic the software, iOS or Android so you can test in a simulated virtual environment.

This is a much more inexpensive solution, but of course, you do get what you pay for.

You do not get to interact with the actual device so if there is a software and hardware compatibility bug, it will be hard to catch in the testing phase which can upset your users after launch.

In addition, since the software is just emulated you are now relying on the provider to accurately simulate the software which they may or may not do 100% right.

Since you are not using physical devices, you lose control of the testing process as well. If the provider is having software issues, it will impact your testing results and potentially delay your app release. If there are problems, you’ll have to rely on the tech support and customer service of the 3rd party. Unfortunately, one of the biggest players in the game—Perfecto Mobile–seems to not have the best reputation when it comes to customer service and have been involved in patent infringement lawsuits.

Emulator Pros

  • Very cheap
  • Easy to get started

Emulator Cons

  • Emulator is not always 100% accurate with software simulation
  • 3rd party has significant control over your testing
  • Many competitors, each has strengths but none all excel at everything
  • You cannot test how software interacts with the hardware

Conclusion

Both emulators and real device solutions have their own strengths and weaknesses but if you can afford it and are serious about mobile device testing, then there’s no substitute for testing on real devices. Of course, an emulator is a great solution for those on a small budget.

So what did you think? Any thoughts on what you prefer? Let us know!

Check out this article if you’d like to see the pros and cons of mobile device testing on emulators and real devices!

 

 

 

 

 

 

 

Mobile Device Testing: Top 5 Testing Tips

Properly testing your mobile devices before any app releases, new website launches or software updates is crucial to weed out the potential bugs. Bugs bug people and cost you customers, visitors and most importantly, time and money. But device testing can be incredibly time and money consuming itself!

That’s why we’ve created a list of the Top 5 Mobile Device Testing Tips so let’s get started!

  1. Do Your Research First

An ounce of prevention is worth a pound of cure. By making sure you have enough data before testing, you can significantly streamline the process.

Where do app crashes usually happen the most? Are 60% of crashes on say the Checkout page? Or maybe they occur the most on a level 5 of your game?

What about types of devices? Are most of your app downloads on tablets or phones? What kinds of tablets and phones? Are 90% of downloads on the iPhone 6s? Or maybe the Samsung S7 Edge?

Do you want your app to target people between the ages of 18 and 24? Then you need to find out what the most common devices are for that demographic and build and test your app on those devices.

By knowing the data, you can establish starting points and priorities as well as most thoroughly testing common pain points and not wasting time on areas that do not typically have bugs.

2. Test Proportionately

After you’ve done your research, you can now spend a proportionate amount of time testing on devices based on your market share or desired market.

For example, if 70% of your app downloads are on Android devices, 25% on Apple devices and 5% on Windows devices then you should spend 70% of your testing time and budget on Android, 25% on Apple and 5% on Windows.

Easy right? By making sure you are most thorough on the devices that have the most downloads, you have a better chance of keeping more people happy. If you miss a bug on Windows devices, you’re only upsetting 5% of your customer base instead of 70%. You can fix bugs on less popular devices as you hear about them while keeping the bulk of your customers and subscribers happy.

You should further refine by device type as well as manufacturer. If 90% of the 70% of Android devices with your app are made by Samsung, then you spend your testing time proportionally there as well.

3. The Right Tools for the Job

PCIt’s important to do both manual and automated app testing and having a tool that can effectively do both makes your life a whole lot easier. Device Connect by Mobile Labs is great for mobile device testing because you can do both manual and automated testing on one machine.

Oh yeah, it’s also connected to real devices too so you can see real-world results instead of simple software that uses an emulator like Perfecto which only simulates the software. Why is that important? Because an emulator leaves out how well an app software interfaces with the hardware!

Not to mention that when it comes time for manual testing, you don’t need to hunt down people with the devices you need. Forget about BYOD testing policies!

4. Focus, Focus, Focus

Once you know where the problems are, focus on one at a time and make surgical fixes, not broad sweeping ones.

Make one code change, record the changes, then test your devices again and record the results.

Not only does this provide valuable insights into what exactly went wrong so you can prevent making the same mistake in the future, it also prevents you from over correcting and causing other issues in other areas of the software.

5. Be Scalable and Think Long Term

Building an app is exciting and it’s easy to get into the mindset of just “launch it and fix things later” but your short term monetary gains will be short lived as you spend money and resources fixing issues and losing subscribers in the long term. The less problems there are at launch, the less time you spend testing in the future.

On top of that, you want to make sure your software has a solid foundation to build on from the very beginning that can support your company goals and initiatives for years to come. If not, then you will need to start over from scratch–which will of course require you to start from scratch when it comes to your mobile device testing.

Like they say, measure twice but only cut once!

Conclusion

So what did you think of our mobile device testing tips? Are they helpful? Do you have others that you would like to share? Let us know in the comments!

Happy testing everyone!