Getting Start With Unit Test for an HTTP REST Application with Python

Using Unittest Framework

Getting Started With Unit Testing for a HTTP REST Application with Python

July 20, 2022

Wasin Waeosri

Developer Advocate

Introduction

Last Updated: Feb 2025

Today, applications are bigger and more complex. A few changes to the source code to add more features or fix bugs can make unexpected behavior in an application. Developers cannot just wait for the test result from the QA team anymore. They need to do unit testing regularly as an integral part of the development process.

Unit testing is a software testing method that helps developers verify if any changes break the code. Unit testing significantly improves code quality, saves time to find software bugs in an early stage of the development lifecycle, and improves deployment velocity. Unit testing is currently the main process of a modern Agile software development practice such as CI/CD (Continuous Integration/Continuous Delivery), TDD (Test-driven development), etc.

Modern applications also need to connect to other services like APIs, databases, data storage, etc. The unit testing needs to cover those modules too. This example project shows how to run unit test cases for a Python application that performs HTTP REST operations which is the most basic task of today's application functionality. With unit testing, developers can verify if their code can connect and consume content via HTTP REST API in any code updates.

The demo application uses a de-facto Requests library to connect to the Delivery Platform (RDP) APIs (formerly known as Refinitiv Data Platform)as the example HTTP REST APIs and uses the Python built-in unittest as a test framework.

Note: This article does not cover all test cases for the HTTP operations and all RDP APIs services. It aims to give the readers an idea about how to unit test an application that makes an HTTP connection with Python only.

Unit Testing Overview

Unit testing is the smallest test that focuses on checking that a single part of the application operates correctly. It breaks an application into the smallest, isolated, testable component called units, and then tests them individually. The unit is mostly a function or method call or procedure in the application source code. Developers and QA can test each unit by sending any data into that unit and see if it functions as intended.

A unit test helps developers to isolate what is broken in their application easier and faster than testing an entire system as a whole. It is the first level of testing done during the development process before integration testing. It is mostly done by the developers automated or manually to verify their code.

You can find more detail about the unit test concept from the following resources:

Python Guide: Testing Your Code article.
How and when to use Unit Testing properly post.

Introduction to Python Unittest framework

The unittest is a Python-built unit testing framework. It supports both test case (the individual unit of testing) and test runner (a special application designed for running test cases and provides the output result).

The unittest framework has the following requirements:

The test cases must be methods of the class.
That class must be defined as a subclass of unittest.TestCase class.
Use a series of special assertion methods in the unittest.TestCase class instead of the built-in assert statement.
The methods' names must start with the letter test as a naming convention for the test runner.
The test cases file name must start with test_ as a naming convention for the test runner.

Example from unittest official page:

To run the test, just run the following command:

Please find more detail about the unittest framework from the following resources:

What is Delivery Platform (RDP) APIs?

The RDP APIs provide various LSEG data and content for developers via easy-to-use Web-based API.

RDP APIs give developers seamless and holistic access to all of the LSEG content such as Environmental Social and Governance (ESG), News, Research, etc, and commingled with their content, enriching, integrating, and distributing the data through a single interface, delivered wherever they need it. The RDP APIs delivery mechanisms are the following:

Request - Response: RESTful web service (HTTP GET, POST, PUT or DELETE)
Alert: delivery is a mechanism to receive asynchronous updates (alerts) to a subscription.
Bulks: deliver substantial payloads, like the end-of-day pricing data for the whole venue.
Streaming: deliver real-time delivery of messages.

This example project is focusing on the Request-Response: RESTful web service delivery method only.

For more detail regarding the RDP APIs, please see the following APIs resources:

Quick Start page.
Tutorials page.

RDP APIs Application Workflow

RDP APIs entitlement check is based on OAuth 2.0 specification. The first step of an application workflow is to get a token from RDP Auth Service, which will allow access to the protected resource, i.e. data REST API.

The API requires the following access credential information:

Username: The username.
Password: Password associated with the username.
Client ID: This is also known as AppKey, and it is generated using an App key Generator. This unique identifier is defined for the user or application and is deemed confidential (not shared between users). The client_id parameter can be passed in the request body or as an “Authorization” request header that is encoded as base64.

Once the authentication success, the function gets the RDP Auth service response message and keeps the following RDP token information in the variables.

access_token: The token used to invoke REST data API calls as described above. The application must keep this credential for further RDP APIs requests.
refresh_token: Refresh token to be used for obtaining an updated access token before expiration. The application must keep this credential for access token renewal.
expires_in: Access token validity time in seconds.

Next, after the application received the Access Token (and authorization token) from RDP Auth Service, all subsequent REST API calls will use this token to get the data. Please find more detail regarding RDP APIs workflow in the following resources:

Project Structure

This example project is a Python console application that login to the RDP platform, then requests the company's Environmental Social and Governance (ESG) data and meta information from the RDP ESG and Search Explore services respectively. The project structure is as follows:

app.py: The main console application.
rdp_controller/rdp_http_controller.py: The main HTTP operations class. This is our focusing class for unit testing.
tests/test_rdp_http_controller.py: The main test cases class that tests all rdp_http_controller.py class's methods. This is our focus test suite in this project.
tests/test_app.py: The test suite class that tests some app.py methods.
tests/fixtures: The test suite fixture/resource files.

Unit Testing RDP APIs Authentication

Let’s start with the class that operates HTTP request-response messages with the RDP services. The rdp_controller/rdp_http_controller.py class uses the Requests library to send and receive data with the RDP HTTP REST APIs. The code for the RDP authentication is shown below.

The rdp_authentication() method above just create the request message payload, and send it to the RDP Auth service as an HTTP Post request. The return values can be as follows

If the authentication success, returns the access_token, refresh_token, and expires_in information to the caller.
If the URL or credentials parameters are empty or none, raise the TypeError exception to the caller.
If the authentication fails, raise the Requests' HTTPError exception to the caller with HTTP status response information.

Let’s leave the rdp_authentication() method there and continue with the test case. The basic test case scenario is to check if the rdp_authentication() method can handle a valid RDP login and empty parameters scenarios.

The test class is tests\test_rdp_http_controller.py file (please noticed a tests folder). It loads the test configurations such as the RDP APIs URLs from a .env.test environment variables file.

The setUpClass() is a class method that is called only once for the whole class before all the tests. The function is useful for setting up a fixture which can be sample data, preconditions states, context, or resources needed to run a test. We initialize the RDPHTTPController object (as app class variable) and the RDP Base URL string (as base_URL class variable) as our fixture here.

Note: The counterpart method of setUpClass() method is tearDownClass() which is called after all tests in the class have run.

The test_login_rdp_success() is a test case for the success RDP Authentication login scenario. It just sends the RDP Auth Service URL and RDP credentials to the rdp_authentication() method and checks the response token information. Please noticed that a unit test just focuses on if the rdp_authentication() returns no empty/zero token information only. The token content validation would be in a system test (or later) phase.

Please see more detail about the supported assertion methods on the unittest framework page.

The test_login_rdp_none_empty_params() is a test case that check if the rdp_authentication() method handles empty or none parameters as expected (throws the TypeError exception and not return token information to a caller).

The example of a test runner result is shown below.

If you input invalid RDP credentials in a .env.test file, a test runner shows the following test failure result (Do not worry, I will cover fail test cases later).

However, the test suite above makes HTTP requests to RDP APIs in every run. It is not a good idea to flood requests to external services every time developers run a test suite when they have updated the code or configurations.

Unit test cases should be able to run independently without relying on external services or APIs. The external dependencies add uncontrolled factors (such as network connection, data reliability, etc) to unit test cases. Those components-to-components testing should be done in an integration testing phase.

So, how can we unit test HTTP request method calls without sending any HTTP request messages to an actual server? Fortunately, developers can simulate the HTTP request and response messages with a mock object.

Mocking Python HTTP API call with Responses

A mock is a fake object that is constructed to look and act like real data within a testing environment. We can simulate the various scenario of the real data with a mock object, then use a mock library to trick the system into thinking that that mock is the real one.

The purpose of mocking is to isolate and focus on the code being tested and not on the behavior or state of external dependencies. By mocking out external dependencies, developers can run tests as often without being affected by any unexpected changes or irregularities of those dependencies. Mocking also helps developers save time and computing resources if they have to test HTTP requests that fetch a lot of data.

This example project uses the Responses library which is built specifically for mocking the Requests library.

Adding a mock Object to the test case

So, I will start with a mock object for testing a successful RDP login case. Firstly, create a rdp_test_auth_fixture.json fixture file with a dummy content of the RDP authentication success response message in a tests/fixtures folder.

Next, load this rdp_test_auth_fixture.json file in the setUpClass() method to a mock_valid_auth_json class variable. The other test cases can use this mock json object for the dummy access token information.

The Responses library lets developers register mock responses to the Requests library and cover the test method with responses.activate decorator. Developers can specify the endpoint URL, HTTP method, status response, response message, etc of that request via a responses.add() method.

Example Code:

The code above set a Responses mock object with the https://api.refinitiv.com/auth/oauth2/v1/token URL and HTTP POST method. The Requests library then returns a mock_valid_auth_json JSON message with HTTP status 200 and Content-Type application/json to the application for all HTTP POST request messages to https://api.refinitiv.com/auth/oauth2/v1/token URL without any network operations between the machine and the actual RDP endpoint.

Testing Invalid RDP Authentication Request-Response

This mock object is also useful for testing false cases such as invalid login too. The test_login_rdp_invalid() method is a test case for the RDP Authentication login failure scenario. We set a Responses mock object for the https://api.refinitiv.com/auth/oauth2/v1/token URL and HTTP POST method with the expected error response message and status (401 - Unauthorized).

Once the rdp_authentication() method is called, the test case verifies if the method raises the requests.exceptions.HTTPError exception with the expected error message and status. The test case also makes assertions to check if the method does not return token information to a caller.

With mocking, a test case never needs to send actual request messages to the RDP APIs, so we can test more scenarios for other RDP services too.

Unit Testing for RDP APIs Data Request

That brings us to requesting the RDP APIs data. All subsequent REST API calls use the Access Token via the Authorization HTTP request message header as shown below to get the data.

Header:
- Authorization = Bearer <RDP Access Token>

Please notice the space between the Bearer and RDP Access Token values.

The application then creates a request message in a JSON message format or URL query parameter based on the interested service and sends it as an HTTP request message to the Service Endpoint. Developers can get RDP APIs the Service Endpoint, HTTP operations, and parameters from RDP's API Playground page - which is an interactive documentation site developers can access once they have a valid RDP account.

The example console application consumes content from the following RDP Services:

ESG Service /data/environmental-social-governance//views/scores-full endpoint that provides full coverage of LSEG proprietary ESG Scores with full history for consumers.
Discovery Search Explore Service /discover/search//explore endpoint that explores LSEG data based on searching options.

However, this development article covers the ESG Service test cases only. The Discovery Search Explore Service's test cases have the same test logic as the ESG's test cases.

Unit Testing HTTP Request Source Code for The RDP ESG Service

Now let me turn to test the Environmental Social and Governance (ESG) service endpoint. The code in the rdp_controller/rdp_http_controller.py class for requesting the ESG data is shown below.

The rdp_request_esg() method above just create the request message payload, and send it to the RDP ESG service as HTTP GET request with the Requests requests.get() method. The return values can be as follows

If the request success (HTTP status is 200), returns the response data in JSON message format.
If the URL or access token, or universe values are empty or none, raise the TypeError exception to the caller.
If the request fails, raise the Requests' HTTPError exception to the caller with HTTP status response information.

Testing a valid RDP ESG Request-Response

The first test case is the request success scenario. I will begin by creating a fixture file with a valid ESG dummy response message. A file is rdp_test_esg_fixture.json in a tests/fixtures folder.

Next, create the test_request_esg() method in the test_rdp_http_controller.py file to test the easiest test case, the successful ESG data request-response scenario. This method creates a mock object for the RDP https://api.refinitiv.com/data/environmental-social-governance/v2/views/scores-full endpoint URL with the HTTP GET method.

When the Requests library receives the HTTP GET request for that URL, it returns a mock ESG data JSON object with HTTP Status 200 and Content-Type application/json to the application. The test_request_esg() method then verifies if the response data is in JSON/Dictionaries type, and checks if the message contains the basic ESG fields.

Testing Requesting RDP ESG data with an expired token

That brings us to one of the most common RDP APIs failure scenarios, applications request data from RDP with an expired access token. The test_request_esg_token_expire() method is the one for testing this case with the RDP ESG Service.

The first step is to create a JSON file tests/rdp_test_token_expire_fixture.json as a test fixture with the following content.

All RDP data services use the same token expire error message, so we load this fixture file as a class variable named mock_token_expire_json to use in all test case methods.

The token expires error message is sent from the RDP services to applications with HTTP error status (401 - As of July 2022), the test_request_esg_token_expire() method simulates the token expire error message and HTTP 401 status with a mock object as follows.

Next, this test case verifies if the rdp_request_esg() method raises the requests.exceptions.HTTPError exception with the expected error message and status.

The other common RDP APIs failure scenario is the application sends the request message to RDP without the access token in the HTTP request header. However, the access token is one of the rdp_request_esg() method required parameters. If the access token is not presented (None or Empty), the method raises the TypeError exception and does not send an HTTP request message to the RDP. The test_request_esg_none_empty() method is the one that covers this test case.

Testing Requesting RDP ESG data with an invalid item

Now we come to other common test cases, what if users request invalid item name data from the RDP ESG service? We can use a mock object to simulate invalid item request-response messages too.

When the ESG Service receives a request message with an invalid item name, it returns the following error response message to the applications as HTTP Status 200 (As of July 2022):

The test_request_esg_invalid_ric() method set a mock invalid item request-response messages to the Requests library, and check if the rdp_request_esg() method returns the error message with HTTP status 200 as expected.

That’s all I have to say about unit testing the Python HTTP code with Requests and Responses libraries.

Prerequisite

This demo project requires the following dependencies.

RDP Access credentials.
Python 3.10 and above
Docker Desktop/Engine application for running the test suite with Docker.
Internet connection.

Please contact your LSEG representative to help you to access the RDP account and services. You can find more detail regarding the RDP access credentials set up from the lease see the Getting Started for User ID section of the Getting Start with Data Platform article.

How to run the example test suite

The first step is to unzip or download the example project folder into a directory of your choice, then set up Python or Docker environments based on your preference.

Run example test suite in Docker

Firstly, start a Docker desktop or Docker engine on your machine.

Then open a console, then go to the project root and run the following command to build a Docker image.

Once this build completes, run a Docker container with the following command:

To stop and delete a Docker container, press Ctrl+C (or run docker stop python_unittest) then run the following command:

That covers how to run an example test suite.

Conclusion and Next Steps

Unit testing is now the mandatory process of a software development lifecycle for both modern and legacy applications. It helps to expose unintentional behaviors of a tiny part of the application quicker than trying to find bugs in a big complex phase. It speeds up the overall feedback loop and improves trust among the project team. Unit testing also helps improves application source code quality, developers have more confidence in refactoring the source code for better performance and cleaner code. As the author of this article, I also have learned a lot from this project. There are a lot of error handlers or code logic that I never think of until I started to write unit test cases.

This article demonstrates the manual unit testing method. However, developers should run unit test cases automatically every time they made changes to the code (or configurations). The most practical technique is running automated unit tests as part of the developers' Continuous Integration/Continuous Delivery (CI/CD) pipeline. Developers can apply the TDD (Test-driven development) approach that writing and correcting the failed tests before writing new code with their project too.

The unittest framework and Responses mocking library are very good starting points to learn a unit test with Python and build a simple test suite to test HTTP operations source code. If developers need more advanced features, they can explore other Python unit test frameworks such as pytest, nose2, or doctest.

At the same time, the RDP APIs provide various LSEG data and content for developers via an easy-to-use Web-based API. The APIs are easy to integrate into any application and platform that supports the HTTP protocol and JSON message format.