Software testing services that drive product quality & customer satisfaction.

Testing - Selenium, Saucelabs, Appium, Mocha, Katalon, SoapUI, Gatling, JMeter, Hoverfly DevOps - Jenkins, CircleCI, TravisCI, Codeship, Gradle

“ You can throw paint against the wall and eventually you might get most of the wall, but until you go up to the wall with a brush, you'll never get the corners. “ We love the metaphor because it applies to testing as well. Choosing the right testing strategy is the same kind of choice you'd make when choosing a brush for painting a wall. Would you use a fine-point brush for the entire wall? Of course not. That would take too long and the end result would probably not look very even. Would you use a roller to paint everything, including around small areas? No way. There are different brushes for different use cases and the same thing applies to tests. A test plan is useful to put discipline into the testing process. Otherwise, it’s often a mess when team members draw different conclusions about the scope, risk and prioritization of product features. The outcomes are better when the entire team is on the same page. In an agile environment, where we work in short sprints or iterations, each sprint is focused on only a few requirements or user stories, so it is natural that documentation may not be as extensive, in terms of both number and content. We should not have an extensive test plan in agile projects for each sprint due to time constraints, but we do require a high-level agile test plans as a guideline for agile teams. The purpose of the agile test plan document is to list best practices and some form of structure that the teams can follow. Remember, agile does not mean unstructured. Test plans have a history of not being read, and for good reason: The typical plan includes 10 to 40 pages of dry technical information, requirements, details on test execution, platforms, test coverage, risk, and test execution calendars and assignments. No wonder people refer to test plans as doorstops. The elements of a concise feature/project test plan Scope: What you’ll test. Objective: The main goal of the client for this project. Out of scope: What you won’t test. Roles and responsibilities: How many QAs (QA lead, Automation tester, QA analyst, etc) and what they’ll do in the project. Methodology: The approach of the project to test, is it BDD?, is it agile? are you doing test cases ? Browsers/OS/Devices to test: This should be defined by the client, however as a QA you need to know the most used browsers/OS/devices worldwide and if the client doesn’t know you can always propose stuff to test, of course, the last word is the client’s word. Types of testing that will be performed (security, performance, automation, accessibility, etc): Not all projects require all testing types, this depends on the project, and sometimes internal testers on the client side do some of this tests, from my perspective, you should always propose to do all testing types if you have the skills to do them. Guidelines for bug reporting: Each company has their own template, it’s good to have it here. Description of bugs severity: You need to describe what’s a blocker, critical, major or minor issue in here so all the team is clear about the bugs the QA team will log. Tools to use: This is good for all the team to agree on tools to use and not change them in the middle of the project, tools are important here because you may not know all tools requested by the client, of course this may change, but still you should add them. Risks: You need to highlight the risks of the project, for instance: deadline too close, training needed for some tool, team is not big enough, etc Environments: This may not be known in this phase of the project, but if you do, don’t forget to add which environments will be use, Release exit criteria Define when the release is good enough to go. For example, you might release only with a 99% pass rate for smoke tests, or when no critical defects have been entered for five days. Describe how you judge when application quality is high enough here. Proposed test scenarios and test coverage can look like this based on Agile Testing Quadrants Unit Testing WHY: To ensure code is developed correctly WHO: Developers / Technical Architects WHAT: All new code + re-factoring of legacy code as well as Javascript unit Testing WHEN: As soon as new code is written WHERE: Local Dev + CI (part of the build) HOW: Automated, Junit, TestNG, PHPUnit API / Service Testing WHY: To ensure communication between components are working WHO: Developers / Technical Architects WHAT: New web services, components, controllers, etc WHEN: As soon as new API is developed and ready WHERE: Local Dev + CI (part of the build) HOW: Automated, Soap UI, Rest Client Acceptance Testing WHY: To ensure customer’s expectations are met are working WHO: Developer / SDET / Manual QA WHAT: Verifying acceptance tests on the stories, verification of features etc WHEN: When the feature is ready and unit tested WHERE: CI / Test Environment HOW: Automated (Cucumber) System Testing / Regression Testing / UAT WHY: To ensure the whole system works when integrated are working WHO: SDET / Manual QA / Business Analyst / Product Owner WHAT: Scenario Testing, User flows and typical User Journeys, Performance and security testing WHEN: When Acceptance Testing is completed WHERE: Staging Environment HOW: Automated (Webdriver) Exploratory Testing

Unit test - A test verifying methods of a single class. Any dependencies external to the class are ignored or mocked out. Note that some single class tests also qualify as feature tests in a few cases, depending on the scope of the “feature” under test. The reason we write unit tests (in general) is to build greater confidence in the code we write. This can be used as a tool to drive the design of our code, a-la-TDD, or at least it can be used to ensure that the code we've written returns an expected output for some input. This also gives us a much greater confidence in performing refactorings on the existing code base, as broken test cases can help us catch any change in class/method APIs, as well as potentially breaking expected return types. We don’t consider Unit tests as 'testing-costs. We think that Unit tests should be part of 'core' engineering & a part of development. Not a task that's added to testing costs. If you aren't writing unit tests (irrespective of whether its TDD or not), you are not developing/engineering your product right. You are only building a stack of cards _hoping_ that it wouldn't collapse at some point in the future. Integration (Feature) test - The meaning of Integration testing is quite straightforward- Integrate/combine the unit tested module one by one and test the behavior as a combined unit. A test covering many classes and verifying that they work together. We normally do Integration testing after “Unit testing”. Once all the individual units are created and tested, we start combining those “Unit Tested” modules and start doing the integrated testing. The end purpose of feature tests is generally much clearer than individual unit tests. A safety net for refactoring - Properly designed feature tests provide comprehensive code coverage and don’t need to be rewritten because they only use public APIs. Attempting to refactor a system that only has single class tests is often painful, because developers usually have to completely refactor the test suite at the same time, invalidating the safety net. This incentivizes hacking, creating tech debt Testing from customer point-of-view - Leads to better user APIs Test end-to-end behavior - With only single class tests, the test suite may pass but the feature may be broken, if a failure occurs in the interface between modules. Feature tests will verify end-to-end feature behavior and catch these bugs. Write fewer tests - A feature test typically covers a larger volume of your system than a single class test. Service as pluggable library - If setup correctly, feature tests lead towards a service design in which the service module itself is embeddable in other applications. Test remote service failure and recovery - It’s much easier to verify major failure conditions and recovery in feature tests, by invoking API calls and checking the response. Our approach to testing (Unit, Integration (Feature), End-to-End) As indicated here, the pyramid shows from bottom to top: Unit, Integration(Feature), E2E. As you can see from the test pyramid, Unit tests are the fastest and cheapest way to ensure software quality. Integration tests are slower but have higher impact on the quality of features delivered. One thing that it doesn't show though is that as you move up the pyramid, the confidence quotient of each form of testing increases. You get more bang for your buck. So while E2E tests may be slower and more expensive than unit tests, they bring you much more confidence that your application is working as intended. Most teams settle for the 70/20/10 rule as promoted by Google Testing which splits testing to 70% unit tests, 20% integration tests, and 10% end-to-end tests. As a team, we want to be able to answer questions like is our application working the way we want it to, or more specifically: Can users log in? Can users send messages? Can users receive notifications? The problem with the 70/20/10 strategy which focuses on unit tests is that it doesn’t answer these questions and many other important high level questions. Our new strategy was to split our automated tests into 40% unit tests, 40% integration tests and 20% end-to-end tests. Our approach is to write more integration tests than Unit tests so that we can focus more on feature quality. Of course, it depends on the solution and our test plan.

TDD stands for Test Driven Development approach. So why TDD? The short answer is “because it is the simplest way to achieve both good quality code and good test coverage”. TDD starts with writing the tests, which fail until the app is written. Provided the tests are accurate, you write the app so that the tests pass. For a simple calculator you will ultimately have one of four (or more) functions, at it's simplest you need add, subtract, divide by and multiply by. The point of TDD is to identify these four functions, not to test them. If you cannot break down your program into these simple one purpose functions then perhaps you need to either rethink your acceptance criteria or you need clearer requirements. TDD's goal isn't *just* to ensure you start with tests, but also (mainly) to simplify your program and keep it on topic. The goal of tests isn't just to find bugs in newly-written code. It's to defend against regressions, and a regression is even harder to localize, given the team may not even be well-familiar with the failing code. Over the years, we’ve learnt to test "what a thing is supposed to do, not how it does it". Usually this would mean to write high level tests or at least to draft up what using the API might end up looking like (be it a REST API or just a library). This approach comes with the benefit that you can focus on designing a nice to use and modular API without worrying on how to solve it from the start. And it tends to produce designs with less leaky abstractions. We prefer to start by writing a final API candidate and its integration tests and only write/derive specific lower level components/unit tests as they become required to advance on the integration test. Our criticism on starting with the bottom-up is that you may end up leaking implementation details in your tests and API because you already defined how the low level components work. So, what is TDD about?It is about defining some steps or a procedure to make sure from day one that your project will abide to automated unit testing concepts and implementations. In TDD developers - Write only enough of a unit test to fail. - Write only enough production code to make the failing unit test pass. What is RGR or Red-Green-Refactor in TDD? The red, green, refactor approach helps developers compartmentalize their focus into three phases: Red — think about what you want to develop. Create a unit test that fails. Green — think about how to make your tests pass. Write production code that makes test pass. Refactor — think about how to improve your existing implementation. Clean up the code. This cycle is typically executed once for every complete unit test, or once every dozen or so cycles of the three laws. The rules of this cycle are simple. Create a unit tests that fails Write production code that makes that test pass. Clean up the mess you just made. The RGR cycle tells us to first focus on making the software work correctly; and then, and only then, to focus on giving that working software a long-term survivable structure.

Whenever developers change or modify their software, even a small tweak can have unexpected consequences. Regression testing is testing existing software applications to make sure that a change or addition hasn’t broken any existing functionality. Its purpose is to catch bugs that may have been accidentally introduced into a new build or release candidate, and to ensure that previously eradicated bugs continue to stay dead. Usually regression testing is done repeatedly, any time a change is made, which makes it a good candidate for automated testing. Two-level approach In agile development, regression should be happening throughout the cycle as part of automated and unit tests, and continuously expanded to cover any new issues that arise as well as new steel-thread stories. We follow and recommend this regression cycle - 1. Iteration regression. The team perform iteration regression at the end of each sprint. Iteration regression specifically focuses on features and changes made in the iteration and areas of the application that could be affected. 2. Full regression. Test engineers run full regression before releases and project milestones to ensure that the application works as planned. Regression testing is a risk management game. Testing everything would be too costly, so we need to focus our efforts. -Identify highest priority test cases - each sprint, prioritize all your test cases, and flag those with the highest priority. These get added to your regression test suite and prioritized in context with the other tests there. -Start the sprint with regression - as testers, we'll typically have little in the way of new work to test at the start of a sprint. In addition to using this time to plan the testing for the current sprint, use it to run regression test cases for previous sprints. If you're starting sprint 10, you would run regression test cases for sprints 1 - 8 (because there's no changes to sprint 9 code yet). In sprint 11 you run test cases for sprints 1 - 9. And so on. -Prioritize aggressively - If the work for the current sprint means we can only cover the ten highest priority regression test cases, then that's what we cover -Document your regression - make sure that part of each sprint review includes what you did not regression-test and why. - Treat your test suites as a backlog - and groom them aggressively. If a test case isn't relevant anymore, don't hesitate to retire it. If one needs updating, update it. - Give your test cases execution time estimates - We estimate how long it takes to run each test case and plan accordingly. - Plan test cases as part of the sprint planning - Your regression test cases are as much a part of the sprint planning as any other activity. Build them into the sprint planning session if at all possible. Having an automated test suite with good coverage is definitely very helpful. However, we wouldn't recommend relying entirely on automated test for regression, as there are some types of bugs that automated tests aren't particularly good at detecting. We start adding automated tests for basic smoke tests flows and build up to the flows for the acceptance tests or even some functional ones in the sensible areas. A time-efficient way of complimenting the automation effort with manual testing is to pick out a subset of manual tests based on a risk analysis of the system. Have a think about the areas of the system that are most likely to be impacted by the changes for the sprint, and only target those areas with your manual testing. We have found this approach results in a much shorter regression testing time towards the end of each sprint, and it has been enough for the level of quality required for our projects. However, it does leave a little room for minor to trivial bugs to slip by in low-risk areas because the low-risk areas are entirely dependent on the automated tests to detect bugs. This may not be acceptable for some project standards, so just be aware of that risk. There are a few approaches to consider. Keep all your test cases, but alternate when you run them. Perhaps run some cases every other release, others get run with each release candidate. Focus regression testing in areas potentially impacted by new feature development. Metrics should determine when you're done. Test until the frequency of finding bugs falls below a given threshold. Regression manual tests just roundup acceptance tests + functional + a few negative ones. Depending on what you are testing, maybe you would need performance & load tests also. For best results, we usually use the manual test cases for those at-risk areas as a guide for my exploratory testing.

Acceptance Test Driven Development (ATDD) aims to help a project team flesh out user stories into detailed Acceptance Tests that, when executed, will confirm whether the intended functionality exists. “By continuously testing for the existence of a given functionality, and writing code to introduce functionality that can pass the Acceptance Tests, developers’ effort is optimised to the point of just meeting the requirement.” ATDD is like BDD in that it requires tests to be created first and calls for the code to be written to pass those tests. However, unlike in TDD where the tests are typically technical-facing unit tests, in ATDD the tests are typically customer-facing acceptance tests. The idea behind ATDD is that user perception of the product is just as important as functionality, so this perception should drive product performance in order to help increase adoption. To bring this idea to life, ATDD collects input from customers, uses that input to develop acceptance criteria, translates that criteria into manual or automated acceptance tests and then develops code against those tests. Like TDD and BDD, ATDD is a test-first methodology, not a requirements driven process. Also like the TDD and BDD methodologies, ATDD helps eliminate potential areas for misunderstanding by removing the need for developers to interpret how the product will be used. ATDD goes one step further than TDD and BDD though because it goes directly to the source (aka the customer) to understand how the product will be used. Ideally, this direct connection should help minimize the need to re-design features in new releases. How is ATDD different from TDD? ATDD borrows from the spirit of Test Driven Development (TDD) in that both techniques allow test cases to be written and executed (and hence fail) before even a single line of code is written. The main difference is that ATDD focuses on testing for business user functionality, while TDD has been traditionally used to run/automate unit tests. In general, TDD is the pioneer that ATDD emulates to fulfil functional testing – however, both the techniques have the same aim: write just enough code, reduce developer efforts, build to detailed requirements and continuously test the product to ensure it meets business user expectations. How is it different from standard Waterfall testing? ATDD is different from standard Waterfall testing because it is a test-first methodology. Standard Waterfall testing calls for test cases to be written upfront based on requirements, whereas ATDD is not a requirements driven testing process. What are best practices? Best practices for testers following an ATDD Agile methodology include: - Interacting closely with customers, for example through focus groups, in order to determine expectations - Leaning on customer-facing team members, such as sales representative, customer service agents and account managers, to understand customer expectations - Developing acceptance criteria based on customer expectations - Prioritizing two questions: - Will customers use the system if it does X? - How can we validate if the system does X?

Microservices attempt to streamline the software architecture of an application by breaking it down into smaller units surrounding the business needs of the application. The benefits that are expected out of doing so include creating systems that are more resilient, easily scalable, flexible, and can be quickly and independently developed by individual sets of smaller teams. This results in a number of benefits over a traditional monolithic architecture such as independent deployability, language, platform and technology independence for different components, distinct axes of scalability and increased architectural flexibility. The challenges of testing microservices Testing microservices is hard. More specifically, end-to-end testing is hard, and that’s something we’ll discuss in greater detail in this article. One important issue we have to keep in mind while working with microservices is API stability and API versioning. To avoid breaking applications depending on a service, we need to make sure we have a solid set of integration tests for microservice APIs and, in case of a breaking change, we have to provide a backwards-compatible way for clients to migrate to a new version at their own pace to avoid large cross-service API change rollouts. Here are a few key challenges associated with testing microservices: Availability: Since different teams may be managing their own microservices, securing the availability of a microservice (or, worse yet, trying to find a time when all microservices are available at once), is tough. Fragmented and holistic testing: Microservices are built to work alone, and together with other loosely coupled services. That means developers need to test every component in isolation, as well as testing everything together. Complexity - There are many microservices that communicate with each other. We need to ensure that every one of them is working properly and is resistant to slow responses or failures from other microservices Performance - Since there are many independent services, it is important to test the whole architecture under traffic close to production. How we test Microservices - Components Tests With Hoverfly In combination with building a service outside-in we also work on component-level testing. This differs from the integration testing in that component testing operates via the public API and tests an entire slice of business functionality. Typically the first wave of component tests utilise the acceptance test scripts, and these assert that we have implemented the business functionality correctly within this service Hoverfly's simulation mode may be especially useful for building component tests. During component tests, we verify the whole microservice without communication over a network with other microservices or external datastores. The following picture shows how such a test is performed for our sample microservice. Contract Tests With Pact The next type of test strategy usually implemented for microservices-based architecture is consumer-driven contract testing. In fact, there are some tools especially dedicated for this type of tests. One of them is Pact. Contract testing is a way to ensure that services can communicate with each other without implementing integration tests. A contract is signed between two sides of communication: consumer and provider. Pact assumes that contract code is generated and published on the consumer side, and then verified by the provider. Pact provides a tool that can store and share the contracts between consumers and providers. It is called Pact Broker. It exposes a simple RESTful API for publishing and retrieving pacts, and an embedded web dashboard for navigating the API. We can easily run Pact Broker on the local machine using its Docker image. Performance Tests With Gatling An important step of testing microservices before deploying them to production is performance testing. Performance testing of a series of core happy paths offered by the service. We typically use JMeter(often triggered via the Jenkins Performance Plugin) or Gatling Gatling is a highly capable load testing tool written in Scala. That means that we also have to use Scala DSL in order to build test scenarios.